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FOREWORD 


This  book  is  one  of  many  technical  management  educational  guides  written  from  a 
Department  of  Defense  (DoD)  perspective;  i.e.,  non-Service  peculiar.  They  are  intended 
primarily  for  use  in  the  courses  at  the  Defense  Systems  Management  College  (DSMC)  and 
secondarily  as  a  desk  reference  for  program  and  project  management  personnel.  These 
guidebooks  are  written  for  current  and  potential  acquisition  management  personnel  who 
are  familiar  with  basic  terms  and  definitions  employed  in  program  offices.  They  are 
designed  to  assist  government  and  industry  personnel  in  executing  their  management 
responsibilities  relative  to  the  acquisition  and  support  of  defense  systems.  They  include: 

a.  Acquisition  Logistics  Guide  (December  1997) 

b.  Mission  Critical  Computer  Resources  Management  Guide  (1990) 

c.  Systems  Engineering  Management  (SEM)  Guide  (January  1990) 

d.  Department  of  Defense  Manufacturing  Management  Handbook  for 
Program  Managers  (April  1989). 

The  objective  of  a  well-managed  T&E  program  is  to  provide  timely  and  accurate  informa¬ 
tion.  This  guide  has  been  developed  to  assist  the  acquisition  community  in  obtaining  a 
better  understanding  of  whom  the  decision  makers  are  and  determining  how  and  when  to 
plan  test  and  evaluation  events. 


John  D.  Claxton 
Professor 

Test  and  Evaluation  Department 
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MODULE 

Management  of 
Test  and  Evaluation 


Test  and  Evaluation  is  a  management  tool 
and  an  integral  part  of  the  development 
process .  This  module  will  address  the  policy 
structure  and  oversight  mechanisms  in 
place  for  test  and  evaluation. 


IMPORTANCE  OF 
TEST  AND  EVALUATION 


1.1  INTRODUCTION 

The  test  and  evaluation  (T&E)  process  is  an 
integral  part  of  the  systems  engineering 
process  which  identifies  levels  of  perfor¬ 
mance  and  assists  the  developer  in  correct¬ 
ing  deficiencies.  It  is  also  a  significant  ele¬ 
ment  in  the  decision-making  process,  pro¬ 
viding  data  supportive  of  trade-off  analy¬ 
sis,  risk  reduction  and  requirements  refine¬ 
ment.  Programmatic  decisions  on  system 
performance  maturity  and  readiness  to 
advance  to  the  next  phase  of  development 
take  into  consideration  demonstrated  per¬ 
formance.  The  issue  of  paramount  impor¬ 
tance  to  the  Service-member  user  is  system 
performance;  i.e.,  will  it  fulfill  the  mission. 
The  test  and  evaluation  process  provides 
data  to  tell  the  user  how  well  the  system  is 
performing  during  development  and  if  it  is 
ready  for  fielding.  The  program  manager 
must  balance  the  risks  of  cost,  schedule  and 
performance  to  keep  the  program  on  track 
to  production  and  fielding.  The  responsi¬ 
bility  of  decision-making  authorities  cen¬ 
ters  on  assessing  risk  tradeoffs.  This  chap¬ 
ter  describes  how  test  and  evaluation  func¬ 
tions  as  a  risk  management  tool.  It  also 
addresses  the  contribution  T&E  makes  by 
providing  empirical  data  before  each  mile¬ 
stone  review. 

1.2  TESTING  AS  A  RISK 
MANAGEMENT  TOOL 

Correcting  defects  in  weapons  has  been 
estimated  to  add  from  10-30  percent  to  the 
cost  of  each  item  (Reference  107).  Such 


costly  redesign  and  modification  efforts 
can  be  reduced  if  carefully  planned  and 
executed  test  and  evaluation  programs  are 
used  to  detect  and  fix  system  deficiencies 
sufficiently  early  in  the  acquisition  process 
(Figure  1-1).  Fixes  instituted  during  Pro¬ 
gram  Definition  and  Risk  Reduction  (PDRR) 
Phase  I,  cost  significantly  less  than  those 
required  in  Engineering  and  Manufactur¬ 
ing  Development  (EMD)  Phase  H,  after  most 
design  decisions  have  been  made. 

Test  and  evaluation  results  figure  promi¬ 
nently  in  the  decisions  reached  at  design 
and  milestone  reviews.  However,  the  fact 
that  T&E  results  are  required  at  major  deci¬ 
sion  points  does  not  presuppose  that  T&E 
results  must  always  be  favorable.  The  final 
decision  responsibility  lies  with  the  deci¬ 
sion  maker  who  must  examine  the  critical 
issues  and  weigh  the  facts.  Only  the  deci¬ 
sion  maker  can  determine  the  weight  and 
importance  that  is  to  be  attributed  to  a 
system's  diverse  capabilities  and  shortcom¬ 
ings  and  the  degree  of  risk  that  can  be 
willingly  accepted.  The  decision-making 
authority  will  be  unable  to  make  this  judg¬ 
ment  without  a  solid  base  of  information 
provided  by  T&E.  Figure  1-2  illustrates  the 
life-cycle  cost  of  the  system  and  how  deci¬ 
sions  impact  program  expenditures. 

A  Defense  Science  Board  1983  Task  Force 
focused  on  the  reduction  of  risk  in  program 
acquisition  (Reference  42).  This  group  made 
the  following  observations: 
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Figure  1-1.  System  Life-Cycle  Technical  Activities 


•  A  poorly-designed  product  cannot  be 

properly  tested  or  produced; 

•  Control  techniques  needed  to  success¬ 
fully  complete  the  design,  test  and  produc¬ 
tion  of  an  item  dictate  the  management 
system  required; 

•  The  industrial  process  of  weapon  sys¬ 
tem  acquisition  demands  a  better  under¬ 
standing  and  implementation  of  basic  en¬ 
gineering  and  manufacturing  disciplines; 

•  The  industrial  process  is  focused  on  the 
design,  test  and  production  of  a  product; 

•  The  design,  test  and  production  pro¬ 
cesses  are  a  continuum  of  interdependent 
disciplines.  Failure  to  perform  well  in  one 
area  will  result  in  failure  to  do  well  in  all 
areas.  When  this  happens,  as  it  does  too 
often,  a  high-risk  program  results  with 
equipment  fielded  later  and  at  far  greater 
cost  than  planned. 

The  Task  Force  developed  a  set  of  templates 
for  use  in  establishing  and  maintaining  low- 
risk  programs.  Each  template  describes  an 
area  of  risk  and  then  specifies  technical  meth¬ 
ods  for  reducing  that  risk.  Program  manag¬ 
ers  and  test  managers  may  wish  to  consult 
these  templates  for  guidance  in  reducing  the 
risks  frequently  associated  with  testprograms. 
Sample  risk  management  templates  were 
publkhed  as  DOD4245.7-M,  'Transition  from 
Development  to  Production." 

1.3  THE  T&E  CONTRIBUTION  AT 
MAJOR  MILESTONES 

Test  and  evaluation  progress  is  monitored 
by  the  Office  of  the  Secretary  of  Defense 
(OSD)  throughout  the  acquisition  process. 
Their  oversight  extends  to  major  defense 
acquisition  programs  or  designated  acqui¬ 
sitions.  Test  and  evaluation  officials  within 
OSD  render  independent  assessments  to 
the  Defense  Acquisition  Board,  the  Defense 


Acquisition  Executive,  and  the  Secretary  of 
Defense  at  each  system  milestone  review. 
These  assessments  are  based  on  the  follow¬ 
ing  T&E  information: 

•  The  Test  and  Evaluation  Master  Plan 
(TEMP)  and  more  detailed  supporting 
documents  developed  by  responsible  Ser¬ 
vice  activities; 

•  Service  test  agency  reports  and  brief¬ 
ings; 

•  Test  and  evaluation,  modeling  and 
simulation,  and  data  from  other  sources 
such  as  Service  program  managers,  labora¬ 
tories,  industry  developers,  studies  and 
analyses. 

At  Milestone  I,  the  OSD  T&E  assessments 
reflect  an  evaluation  of  system  concepts 
and  alternatives  using  early  performance 
parameter  objectives  and  thresholds  found 
in  an  approved  preliminary  TEMP.  At  Mile¬ 
stone  II,  assessments  include  an  evaluation 
of  previously  established  test  plans  and 
test  results.  At  Milestone  III,  assessments 
include  consideration  of  the  operational 
effectiveness  and  suitability  evaluations  of 
weapon  systems. 

A  primary  contribution  made  by  T &E  is  the 
detection  and  reporting  of  deficiencies  that 
may  adversely  impact  the  performance  ca¬ 
pability  or  availability/supportability  of  a 
system.  A  deficiency  reporting  process  is 
used  throughout  the  acquisition  process  to 
report,  evaluate  and  track  system  deficien¬ 
cies  and  to  provide  the  impetus  for  correc¬ 
tive  actions. 

1.3.1  T&E  Contributions  Prior 
to  Milestone  I 

During  Concept  Exploration  (Phase  0)  prior 
to  Milestone  I,  laboratory  testing  and  mod¬ 
eling  and  simulations  are  conducted  by  the 
contractors  and  the  development  agency  to 
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Figure  1-2.  Life-Cycle-Cost  Decision  Impact  and  Expenditures 


demonstrate  and  assess  the  capabilities  of 
key  subsystems  and  components.  The  test 
and  simulation  designs  are  based  on  the 
operational  needs  documented  in  the  Mis¬ 
sion  Need  Statement  and  draft  Operational 
Requirements  Document.  Studies,  analy¬ 
ses,  simulation  and  test  data  are  used  by  the 
development  agency  to  explore  and  evalu¬ 
ate  alternative  concepts  proposed  to  satisfy 
the  user' s  needs.  Also  during  this  period, 
the  operational  test  agency  (OTA)  moni¬ 
tors  concept  exploration  activities  to  gather 
information  for  future  T&E  planning  and 
to  provide  effectiveness  and  suitability  in¬ 
put  desired  by  the  program  manager.  The 
OTA  also  conducts  early  operational  as¬ 
sessments,  as  feasible,  to  assess  the  opera¬ 
tional  impact  of  candidate  technical  ap¬ 
proaches  and  to  assist  in  selecting  preferred 
alternative  system  concepts. 

Toward  the  end  of  the  phase,  the  develop¬ 
ment  agency  prepares  the  Development 
Test  and  Evaluation  (DT&E)  Phase  Report. 
This  report  records  and  presents  T&E  re¬ 
sults  of  system  design(s)  engineering  and 
performance  evaluations.  This  information 
is  incorporated  into  the  Program  Manager's 
Status  Briefing  and  key  documents  that 
form  the  basis  for  the  Milestone  I  decision 
to  proceed  to  Phase  I. 

1.3.2  T&E  Contributions  Prior  to 
Milestone  II 

During  Program  Definition  and  Risk  Re¬ 
duction  (Phase  I)  prior  to  Milestone  II,  con¬ 
cepts  approved  for  prototyping  form  the 
baseline  that  is  used  for  detailed  test  plan¬ 
ning. 

The  development  agency  conducts  devel¬ 
opment  test  and  evaluation  during  Phase  I 
to  assist  with  engineering  design,  system 
development,  risk  identification  and  to 
evaluate  the  contractor's  ability  to  attain 
desired  technical  performance  in  system 


specifications  and  achieve  program  objec¬ 
tives.  The  DT&E  includes  T&E  of  compo¬ 
nents,  subsystems  and  prototype  develop¬ 
ment  models.  Test  and  evaluation  of  func¬ 
tional  compatibility,  interoperability  and 
integration  with  fielded  and  developing 
equipment  and  systems  is  also  included. 
During  this  phase  of  testing,  adequate 
DT&E  is  accomplished  to  ensure  engineer¬ 
ing  is  reasonably  complete  (including  sur¬ 
vivability/  vulnerability,  compatibility, 
transportability,  interoperability,  reliabil¬ 
ity,  maintainability,  safety,  human  factors, 
and  logistic  supportability).  Also,  this  phase 
confirms  that  all  significant  design  prob¬ 
lems  have  been  identified  and  solutions  to 
these  problems  are  in  hand. 

The  Service  operational  test  and  evalua¬ 
tion  (OT&E)  agency  conducts  Early  Op¬ 
erational  Assessments  (EOA)  to  estimate 
the  system's  potential  to  be  operationally 
effective  and  suitable;  identifies  needed 
modifications;  and  provides  information 
on  tactics,  doctrine,  organization  and  per¬ 
sonnel  requirements.  The  early  OT&E 
program  is  accomplished  in  an  environ¬ 
ment  containing  limited  operational  real¬ 
ism.  Typical  operational  and  support  per¬ 
sonnel  are  used  to  obtain  early  estimates 
of  the  user's  capability  to  operate  and 
maintain  the  system.  Some  of  the  most 
important  products  of  user  assessments 
of  system  maintainability  and  support- 
ability  are  human  factors  and  safety  is¬ 
sues. 

The  development  agency  prepares  a  phase 
report  on  the  results  of  Phase  I  DT&E  for 
review  by  the  Service  headquarters  and  the 
Service  acquisition  review  council  prior  to 
system  acquisition  review  by  the  Depart¬ 
ment  of  Defense  (DoD).  The  report  includes 
the  results  of  testing  and  supporting  infor¬ 
mation,  conclusions  and  recommendations 
for  further  engineering  development.  At  the 
same  time,  the  OT&E  agency  prepares  an 
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independent  Early  Operational  Assessment, 
which  contains  estimates  of  the  system's  po¬ 
tential  operational  effectiveness  and  suitabil¬ 
ity.  The  EOA  provide  a  permanent  record  of 
OT&E  events,  an  audit  trail  of  OT&E  data, 
test  results,  conclusions  and  recommenda¬ 
tions.  This  information  is  used  to  prepare  for 
Milestone  n  and  supports  a  recommendation 
of  which  alternative  system  studied  in  Phase 
I  should  proceed  into  engineering  and  manu¬ 
facturing  development. 

1.3.3  T&E  Contributions  Prior  to 
Milestone  III 

Prior  to  Milestone  III,  the  objective  of  Engi¬ 
neering  and  Manufacturing  Development 
(Phase  II)  is  to  design,  fabricate  and  test  a 
preproduction  system  that  closely  approxi¬ 
mates  the  final  product.  Test  and  evalua¬ 
tion  activities  of  the  engineering  develop¬ 
ment  model  during  this  period  yield  much 
useful  information.  For  example,  data  ob¬ 
tained  during  EMD  test  and  evaluation  can 
be  used  to  assist  in  evaluating  the  system's 
maintenance  training  requirements  and 
the  proposed  training  program.  Test  re¬ 
sults  generated  during  EMD  test  and 
evaluation  also  support  the  user  in  refin¬ 
ing  and  updating  employment  doctrine 
and  tactics. 

During  Phase  II  development,  test  and 
evaluation  is  conducted  to  satisfy  the  fol¬ 
lowing  objectives: 

(1)  As  specified  in  program  documents, 
assess  the  critical  technical  issues: 

(a)  Determine  how  well  the  develop¬ 
ment  contract  specifications  have  been  met; 

(b)  Identify  system  technical  deficiencies 
and  focus  on  areas  for  corrective  actions; 

(c)  Determine  whether  the  system  is  com¬ 
patible,  interoperable,  and  canbe  integrated 


with  existing  and  planned  equipment  or 
systems; 

(d)  Estimate  the  reliability,  maintainabil¬ 
ity  and  availability  of  the  system  after  it  is 
deployed; 

(e)  Determine  whether  the  system  is  safe; 
ready  for  Low  Rate  Initial  Production  and  / 
or  lOT&E; 

(f)  Evaluate  effects  on  performance  of 
any  configuration  changes  caused  by  cor¬ 
recting  deficiencies,  modifications  or  prod¬ 
uct  improvements; 

(g)  Assess  human  factors  and  identify 
limiting  factors; 

(2)  Assess  the  technical  risk  and  evaluate 
the  tradeoffs  among  specifications,  opera¬ 
tional  requirements,  life-cycle  costs  and 
schedules; 

(3)  Assess  the  survivability,  vulnerability 
and  logistic  supportability  of  the  system; 

(4)  Verify  the  accuracy  and  completeness 
of  the  technical  documentation  developed 
to  maintain  and  operate  the  weapons  sys¬ 
tem; 

(5)  Gather  information  for  training  pro¬ 
grams  and  technical  training  materials 
needed  to  support  the  weapon  system; 

(6)  Provide  information  on  environmen¬ 
tal  issues  for  use  in  preparing  environmen¬ 
tal  impact  assessments; 

(7)  Determine  system  performance  limi¬ 
tations  and  safe  operating  parameters; 

(8)  Using  Live  Fire  Test  (LFT),  evaluate 
vulnerability  or  lethality  of  a  weapon  sys¬ 
tem  as  appropriate  and  as  required  by 
law. 
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Initial  Operational  Test  and  Evaluation  is 
conducted  prior  to  the  production  decision 
at  Milestone  III  to: 

(1)  Estimate  the  operational  effective¬ 
ness  and  suitability  of  the  system; 

(2)  Identify  operational  deficiencies; 

(3)  Evaluate  changes  in  production  con¬ 
figuration; 

(4)  Provide  information  for  developing 
and  refining  logistics  support  requirements 
for  the  system  and  training,  tactics,  tech¬ 
niques  and  doctrine; 

(5)  Provide  information  to  refine  op¬ 
eration  and  support  (O&S)  cost  estimates 
and  identify  system  characteristics  or  de¬ 
ficiencies  that  can  significantly  impact 
O&S  costs; 

(6)  Determine  whether  the  technical  pub¬ 
lications  and  support  equipment  are  ad¬ 
equate;  in  the  operational  environment. 

Thus,  T&E  activities  intensify  during  Phase 
II  and  make  significant  contributions  to  the 
overall  acquisition  decision  process. 

1.3.4  T&E  Contributions  After 
The  Production  Decision 

After  Milestone  III,  when  the  full  rate  pro¬ 
duction  decision  is  normally  made,  T&E 
activities  continue  to  provide  important 
insights.  Tests  described  in  the  TEMP  but 
not  conducted  during  Phase  II  are  com¬ 
pleted  diu-ing  Phase  HI.  The  residual  DT&E 
may  include  extreme  weather  testing  and 
testing  corrected  deficiencies.  System  ele¬ 
ments  are  integrated  into  the  final  opera¬ 
tional  configuration,  and  development  test¬ 
ing  is  completed  when  all  system  perfor¬ 
mance  requirements  are  met.  During  Phase 
III,  government  representatives  normally 


monitor  or  conduct  the  production  accep¬ 
tance  test  and  evaluation  (PAT&E).  Each 
system  is  verified  by  PAT&E  for  compli¬ 
ance  with  the  requirements  and  specifica¬ 
tions  of  the  contract. 

Postproduction  testing  requirements  may 
result  from  an  acquisition  strategy  calling 
for  block  changes  to  accommodate  accu¬ 
mulated  engineering  changes  or  the  appli¬ 
cation  of  preplanned  product  improve¬ 
ments  (P3I).  This  will  allow  parallel  devel¬ 
opment  of  high-risk  technology  and  modu¬ 
lar  insertion  of  system  upgrades  into  pro¬ 
duction  equipment.  Technology  break¬ 
throughs  and  significant  threat  changes 
may  require  system  modifications.  The  de¬ 
velopment  of  the  modifications  will  require 
development  testing;  and,  if  system  perfor¬ 
mance  is  significantly  changed,  operational 
testing  may  be  appropriate. 

Operational  T&E  activities  continue  after 
the  production  decision  in  the  form  of  Fol- 
low-on  Operational  Test  and  Evaluation 
(FOT&E).  The  initial  phase  of  FOT&E  may 
be  conducted  by  either  the  OT&E  agency  or 
user  commands,  depending  on  Service  di¬ 
rectives.  It  verifies  the  operational  effec¬ 
tiveness  and  suitability  of  the  production 
system,  determines  if  deficiencies  identi¬ 
fied  during  the  lOT&E  have  been  corrected, 
and  evaluated  areas  not  tested  during  lOT&E 
due  tosystem  limitations.  AdditionalFOT&E 
may  be  conducted  over  the  life  of  the  system 
to  refine  doctrine,  tactics,  techniques  and  train¬ 
ing  programs  and  evaluate  future  modifica¬ 
tions  and  upgrades. 

The  OT&E  agency  prepares  a  final  report  at 
the  conclusion  of  each  FOT&E.  This  report 
records  test  results,  describes  the  evalua¬ 
tion  accomplished  to  satisfy  critical  issues 
and  objectives  established  for  FOT&E  and 
documents  its  assessment  of  deficiencies 
resolved  after  EMD.  Deficiencies  that  are 
not  corrected  are  recorded. 
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A  final  report  on  FOT&E  may  also  be  pre¬ 
pared  by  the  using  command  test  team 
emphasizing  the  operational  utility  of  the 
system  when  operated,  maintained  and 
supported  by  operational  personnel  us¬ 
ing  the  concepts  specified  for  the  system. 
Specific  attention  is  devoted  to  the  fol¬ 
lowing: 

(1)  The  degree  to  which  the  system  ac¬ 
complishes  its  missions  when  employed 
by  operational  personnel  in  a  realistic  sce¬ 
nario  with  the  appropriate  organization, 
doctrine,  threat  (including  countermea¬ 
sures  and  nuclear  threats),  environment 
and  using  tactics  and  techniques  devel¬ 
oped  during  earlier  FOT&E; 

(2)  The  degree  to  which  the  system  can  be 
placed  in  operational  field  use,  with  spe¬ 
cific  evaluations  of  availability,  compat¬ 
ibility,  transportability,  interoperability, 
reliability,  wartime  usage  rates,  maintain¬ 
ability,  safety,  human  factors,  manpower 
supportability ,  logistics  supportability  and 
training  requirements; 

(3)  The  conditions  under  which  the  sys¬ 
tem  was  tested  including  the  natural 
weather  and  climatic  conditions,  terrain 
effects,  battlefield  disturbances  and  enemy 
threat  conditions; 

(4)  The  ability  of  the  system  to  perform 
its  required  functions  for  the  duration  of  a 
specified  mission  profile; 

(5)  System  weaknesses  such  as  the  vul¬ 
nerability  of  the  system  to  exploitation  by 
countermeasures  techniques  and  the  prac¬ 
ticality  and  probability  of  an  adversary 
exploiting  the  susceptibility  of  a  system  in 
combat. 

A  specific  evaluation  of  the  personnel  and 
logistics  changes  needed  for  the  effective 
integration  of  the  system  into  the  user's 
inventory  is  also  made.  These  assessments 


provide  essential  input  for  the  later  ac¬ 
quisition  phases  of  the  system  develop¬ 
ment  cycle. 

1.4  SUMMARY 

"Risk  management  is  the  means  by  which 
the  program  areas  of  vulnerability  and  con¬ 
cern  are  identified  and  managed."  (Refer¬ 
ence  20).  Test  and  evaluation  is  the  disci¬ 
pline  that  helps  to  illuminate  those  areas  of 
vulnerability.  The  importance  of  T&E  in 
the  acquisition  process  is  summarized  well 
in  a  December  1986  report  produced  by  the 
General  Accounting  (Dffice  (NSIAD  87-57). 
While  the  following  remarks  focus  on 
OT&E,  they  also  serve  to  underscore  the 
importance  of  the  T&E  process  as  a  whole: 

OT&E  is  the  primary  means  of  assess¬ 
ing  weapon  system  performance. 
OT&E  results  are  important  in  making 
key  decisions  in  the  acquisition  pro¬ 
cess,  especially  the  decision  to  proceed 
from  development  to  production. 
OT&E  results  provide  an  indication  of 
how  well  new  systems  will  work  and 
can  be  invaluable  in  identifying  inef¬ 
fective  or  unreliable  systems  before 
they  are  produced. 

Starting  production  before  adequate 
OT&E  is  completed  has  some  risks.  If 
adequate;,  OT&E  is  not  done  and  the 
weapon  system  does  not  perform  sat¬ 
isfactorily  in  the  field,  significant 
changes  may  be  required.  Moreover, 
the  changes  will  not  be  limited  to  a  few 
developmental  models,  but  may  also 
be  applied  to  items  already  produced 
and  deployed.  In  extreme  situations, 
DoD  also  risks  (1)  deploying  systems, 
which  cannot  adequately  perform  sig¬ 
nificant  portions  of  their  missions,  thus 
degrading  our  deterrent/ defensive  ca¬ 
pabilities  and  (2)  endangering  the 
safety  of  military  personnel  who  oper¬ 
ate  and  maintain  the  systems. 
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2 

THE  TEST  AND  EVALUATION 
PROCESS 


2.1  INTRODUCTION 

The  fundamental  purpose  of  test  and  evalu¬ 
ation  (T&E)  in  a  defense  system's  develop¬ 
ment  and  acquisition  program  is  to  identify 
the  areas  of  risk  to  be  reduced  or  elimi¬ 
nated.  During  the  early  phases  of  develop¬ 
ment,  T&E  is  conducted  to  demonstrate  the 
feasibility  of  conceptual  approaches,  evalu¬ 
ate  design  risk,  identify  design  alternatives, 
compare  and  analyze  tradeoffs,  and  esti¬ 
mate  satisfaction  of  operational  require¬ 
ments.  As  a  system  undergoes  design  and 
development,  the  iterative  process  of  test¬ 
ing  moves  gradually  from  development 
test  and  evaluation  (DT&E),  which  is  con¬ 
cerned  chiefly  with  attainment  of  engineer¬ 
ing  design  goals,  to  operational  test  and 
evaluation  (OT&E),  which  focuses  on  ques¬ 
tions  of  operational  effectiveness,  suitabil¬ 
ity  and  survivability.  Although  there  are 
usually  separate  development  and  opera¬ 
tional  test  events,  DT&E  and  OT&E  are  not 
necessarily  serial  phases  in  the  evolution  of 
a  weapon  system  development.  Combined 
or  concurrent  development  test  and  opera¬ 
tional  test  is  encouraged  when  appropriate 
(possible  cost/ time  savings)  (Reference  16). 

Test  and  evaluation  has  its  origins  in  the 
testing  of  hardware.  This  tradition  is  heavily 
embedded  in  its  vocabulary  and  proce¬ 
dures.  The  advent  of  software-intensive 
systems  has  brought  new  challenges  and 
new  approaches  to  testing,  which  are  dis¬ 
cussed  in  Chapter  17  of  this  management 
guide.  Remaining  constant  throughout  the 
T&E  process,  whether  testing  hardware  or 


software,  is  the  need  for  thorough,  logical, 
systematic  and  early  test  planning  includ¬ 
ing  feedback  of  well-documented  and  un¬ 
biased  T&E  results  to  system  developers, 
users  and  decision  makers. 

Test  and  evaluation  has  many  useful  func¬ 
tions  and  provides  information  to  many 
customers.  The  T&E  gives  information  to: 
developers  for  identifying  and  resolving 
technical  difficulties;  decision  makers  re¬ 
sponsible  for  procuring  a  new  system  and 
for  the  best  use  of  limited  resources;  and  to 
operational  users  for  refining  requirements 
and  supporting  development  of  effective 
tactics,  doctrine  and  procedures. 

2.2  DEFENSE  SYSTEM 
ACQUISITION  PROCESS 

The  defense  system  acquisition  process  was 
revised  in  1996  to  make  it  less  costly,  less 
time-consuming  and  more  responsive  to 
the  needs  of  the  operational  community. 
As  it  is  now  structured,  the  defense  system 
life  cycle  consists  of  the  following  four 
phases: 

(0)  Concept  Exploration 

(I)  Program  Definition  and  Risk  Reduc¬ 
tion 

(II)  Engineering  and  Manufacturing  De¬ 
velopment 
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(Ill)  Production,  Fielding/Deployment 
and  Operational  Support. 

As  Figure  2-1  shows,  these  phases  are  sepa¬ 
rated  by  key  decision  points  when  a  Mile¬ 
stone  Decision  Authority  (MDA)  reviews  a 
program  and  authorizes  advancement  to 
the  next  phase  in  the  cycle.  Thus  T&E  plan¬ 
ning  and  test  results  play  an  important  part 
in  the  milestone  review  process. 

The  following  brief  description  of  the  de¬ 
fense  system  acquisition  process  shows  how 
T&E  fits  within  the  context  of  the  larger 
process.  The  description  is  based  primarily 
upon  information  found  in  Department  of 
Defense  (DoD)  5000. 2-R . 

2.2.1  Concept  Exploration  (Phase  0) 

The  defense  system  acquisition  process 
begins  with  the  submission  of  a  Mission 
Need  Statement.  A  Concept  Exploration 
Phase  follows  the  Milestone  0  decision  dur¬ 
ing  which  alternative  approaches  for  satis¬ 
fying  the  user's  needs  are  investigated.  The 
Concept  Exploration  Phase  concludes  with 
the  Milestone  I  selection  of  a  concept  or 
concepts  to  enter  a  Program  Definition  and 
Risk  Reduction  Phase.  The  Milestone  I  de¬ 
cision  establishes  broad  objectives  for  pro¬ 
gram  cost,  schedule,  and  technical  perfor¬ 
mance.  Key  documents  for  the  T&E  man¬ 
ager  at  the  time  of  the  Milestone  I  review 
are  the  Acquisition  Decision  Memorandum 
(ADM)  (exit  criteria).  Operational  Require¬ 
ments  Document  (ORD),  Acquisition  Strat¬ 
egy,  System  Threat  Assessment  (ST A),  and 
the  Test  and  Evaluation  Master  Plan 
(TEMP).  Additional  program  management 
documents  prepared  before  Milestone  I 
include:  the  Analysis  of  Alternatives  (AO  A), 
Independent  Cost  Estimate,  and  Concept 
Baseline,  Acquisition  Program  Basesline 
(APB),  which  summarizes  the  weapon's 
functional  specifications,  performance  pa¬ 
rameters,  and  cost  and  schedule  objectives. 


2.2.2  Program  Definition 
and  Risk  Reduction  (Phase  I) 

After  the  Milestone  I  decision  for  a  pro¬ 
gram  start,  the  Program  Definition  and 
Risk  Reduction  Phase  begins  during  which 
selected  concepts,  typically  brassboard  or 
early  prototype,  are  refined  through  engi¬ 
neering  and  analysis.  This  phase  ends  with 
the  Milestone  II  decision  to  either  enter  into 
Engineering  and  Manufacturing  Develop¬ 
ment  (EMD)  or  terminate  the  program.  The 
Milestone  11  decision  establishes  more  spe¬ 
cific  cost,  schedule,  and  performance  objec¬ 
tives  and  thresholds.  The  program  office 
for  major  programs  must  give  consider¬ 
ation  to  requesting  a  waiver  for  full-up 
system  level  Live  Fire  Testing  and  identifi¬ 
cation  of  Low  Rate  Initial  Production  (LRIP) 
quantities  for  Initial  Operational  Test  and 
Evaluation  (lOT&E).  Documents  interest¬ 
ing  to  the  T&E  manager  at  the  time  of  the 
Milestone  II  review  include  the  ADM  (exit 
criteria),  updated  TEMP,  updated  STA, 
AOA,  updated  ORD,  Development 
Baseline,  Development  Testing  Phase  re¬ 
port  and  the  Early  Operational  Assessment. 

2.2.3  Engineering  and  Manufacturing 
Development  (Phase  II) 

During  the  EMD  Phase,  the  selected  sys¬ 
tem  and  its  principal  items  of  support  are 
fabricated  as  engineering  development 
models.  Test  articles  normally  are  subjected 
to  qualification  testing,  full-up  Live  Fire 
Testing  and  lOT&E.  This  phase  ends  with 
the  Milestone  III  decision  to  enter  full-rate 
production  of  the  system.  Key  documents 
for  the  T&E  manager  at  the  time  of  the 
Milestone  III  review  are  the  updated  TEMP, 
Development  Testing  Phase  report,  the 
lOT&E  report.  Beyond  LRIP  Report,  and 
Live  Fire  Test  Report.  For  ACAT  I  and 
designated  oversight  programs,  the  Direc¬ 
tor  of  OT&E  (DOT&E)  is  required  by  law  to 
document  his  assessment  of  the  adequacy 
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Figure  2-1 .  Testing  and  the  Acquisition  Process 


of  OT&E  and  the  reported  operational  ef¬ 
fectiveness  and  suitability  of  the  system. 
This  is  done  in  the  Beyond  LRIP  (BLRIP) 
Report.  Also  mandated  by  law  is  the  re¬ 
quirement  for  the  IDOT&E  to  submit  the 
Live  Fire  Test  Report  prior  to  the  program 
proceeding  beyond  LRIP.  These  DOT&E 
Reports  may  be  submitted  as  a  single  as¬ 
sessment. 

2.2.4  Production,  Fielding/Deployment 
and  Operational  Support  (Phase  III) 

The  Milestone  HI  decision  is  followed  by 
Full-Rate  Production  and  Fielding/Deploy¬ 
ment  of  the  system.  This  phase  may  include 
a  major  modification  to  the  production  con¬ 
figuration.  Approval  for  major  modifica¬ 
tions  should  identify  the  actions  and  re¬ 
sources  needed  to  achieve  and  maintain 
operational  readiness  and  support  objec¬ 
tives.  High  cost  may  require  initiation  of 
the  modification  as  a  new  program.  To 
determine  whether  major  upgrades/ modi¬ 
fications  are  necessary  or  deficiencies  war¬ 
rant  consideration  of  replacement,  the  MD  A 
may  review  the  impact  of  proposed  changes 
on  system  operational  effectiveness,  suit¬ 
ability  and  readiness. 

2.3  T&E  AND  THE  SYSTEMS 
ENGINEERING  PROCESS 

In  the  early  1970s,  DoD  test  policy  be¬ 
came  more  formalized  and  placed  greater 
emphasis  on  T&E  as  a  continuing  func¬ 
tion  throughout  the  acquisition  cycle. 
These  policies  stressed  the  use  of  T&E  to 
reduce  acquisition  risk  and  provide  early 
and  continuing  estimates  of  system  op¬ 
erational  effectiveness  and  operational 
suitability.  To  meet  these  objectives,  ap¬ 
propriate  test  activities  had  to  be  fully 
integrated  into  the  overall  development 
process.  From  a  systems  engineering  per¬ 
spective,  test  planning,  testing  and  analy¬ 
sis  of  test  results  are  integral  parts  of  the 


basic  product  definition  process. 

Systems  engineering  has  been  defined  in 
the  DoD  context:  Systems  engineering  is  an 
interdisciplinary  approach  to  evolve  and 
verify  an  integrated  and  optimally  balanced 
set  of  product  and  process  designs  that 
satisfy  user  needs  and  provide  information 
for  management  decision  making  (Figure 
2-2). 

A  system's  life  cycle  begins  with  the  user's 
needs,  which  are  expressed  as  constraints, 
and  the  required  capabilities  needed  to  sat¬ 
isfy  mission  objectives.  Systems  engineer¬ 
ing  is  essential  in  the  earliest  planning  pe¬ 
riod,  in  conceiving  the  system  concept  and 
defining  performance  requirements  for 
system  elements.  As  the  detailed  design  is 
prepared,  systems  engineers  ensure  bal¬ 
anced  influence  of  all  required  design  spe¬ 
cialties,  including  'testability.'  They  resolve 
interface  problems,  perform  design  reviews, 
perform  trade-off  analyses  and  assist  in 
verifying  performance. 

The  days  when  one  or  two  individuals 
could  design  a  complex  system,  espe¬ 
cially  a  huge,  modern-age  weapon  sys¬ 
tem,  are  in  the  past.  Now  systems  are  too 
complex  for  a  small  number  of  general¬ 
ists  to  accommodate;  they  require  too 
much  in-depth  knowledge  over  a  broad 
range  of  areas  and  technical  disciplines. 
System  engineers  coordinate  the  many 
specialized  engineers  involved  in  the  con¬ 
current  engineering  process  through  in¬ 
tegrated  product  and  process  develop¬ 
ment.  Integrated  Product  Teams  (IPT)  are 
responsible  for  the  integration  of  the  com¬ 
ponents  into  a  system. 

Through  interdisciplinary  integration,  sys¬ 
tems  engineering  manages  the  progress  of 
product  definition  from  system  level  to 
configuration-item  level,  detailed  level, 
deficiency  correction,  and  modifications/ 
product  improvements.  Test  results  provide 
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Figure  2-2.  The  Systems  Engineering  Process 


feedback  to  analyze  the  design  progress 
toward  performance  goals.  Tools  of  sys¬ 
tems  engineering  include  design  reviews, 
configuration  management,  simulation, 
technical  performance  measurement,  trade¬ 
off  analysis  and  specifications. 

What  happens  during  systems  engineer¬ 
ing?  It  determines  what  specialists  are  re¬ 
quired,  what  segments  and  nondevelop- 
mental  items  are  used,  design  performance 
limits,  trade-off  criteria,  how  to  test,  when 
to  test,  how  to  document  (specifications), 
and  what  management  controls  to  apply 
(technical  performance  measurement  and 
design  reviews). 

Development  testing  (DT)  and  operational 
testing  (OT)  support  the  technical  reviews 
by  providing  feedback  to  the  systems  engi¬ 
neering  process.  More  information  on  the 
reviews  is  contained  in  Chapter  8. 

2.3.1  The  Systems  Engineering  Process 

The  systems  engineering  process  is  the  it¬ 
erative  logical  sequence  of  analysis,  de¬ 
sign,  test  and  decision  activities  that  trans¬ 
forms  an  operational  need  into  the  descrip¬ 
tions  required  for  production  and  fielding 
of  all  operational  and  support  system  ele¬ 
ments.  This  process  consists  of  four  activi¬ 
ties.  They  include  requirements  analysis, 
functional  analysis  and  allocation,  synthe¬ 
sis,  and  verification  of  performance  (test 
and  evaluation)  which  support  decisions 
on  tradeoffs  and  formalize  the  description 
of  system  elements. 

The  requirements  analysis  activity  is  a  pro¬ 
cess  used  by  the  program  office,  in  concert 
with  the  user,  to  establish  and  refine  opera¬ 
tional  and  design  requirements  that  re¬ 
sult  in  the  proper  balance  between  per¬ 
formance  and  cost  within  affordability 
constraints.  Requirements  analysis  shall 
be  conducted  iteratively  with  functional 


analysis /allocation  to  develop  and  refine 
system  level  functional  and  performance 
requrements,  external  interfaces,  and  pro¬ 
vide  traceability  among  user  requirements 
and  design  requirements. 

The  functional  analysis  activity  identifies 
what  the  system,  component  or  part  must 
do.  It  normally  works  from  the  top  down¬ 
ward  ensuring  requirements  traceability 
and  examining  alternative  concepts.  This  is 
done  without  assuming  how  functions  will 
be  accomplished.  The  product  is  a  series  of 
alternative  Functional  Flow  Block  Diagrams 
(FFBD).  A  functional  analysis  can  be  ap¬ 
plied  at  every  level  of  development.  At 
the  system  level,  it  may  be  a  contractor  or 
Service  effort.  During  Phase  0,  develop¬ 
mental  testers  assist  the  functional  analysis 
activity  to  help  determine  what  each 
component's  role  will  be  as  part  of  the 
system  being  developed.  Performance  re¬ 
quirements  are  allocated  to  system  compo¬ 
nents. 

The  synthesis  activity  involves  invention 
— conceiving  ways  to  do  each  FFBD  task — 
to  answer  the  "how"  question.  Next,  the 
physical  interfaces  implied  by  the  "how" 
answers,  are  carefully  identified  (topologi¬ 
cal  or  temporal).  The  answers  must  reflect 
all  technology  selection  factors.  Synthesis 
tools  include  Requirements  Allocation 
Sheets  (RAS),  which  translate  functional 
statements  into  design  requirements  and 
permit  a  long  and  complex  interactive  in¬ 
vention  process  with  control,  visibility  and 
requirements  traceability.  Developmental 
testers  conduct  prototype  testing  to  deter¬ 
mine  how  the  components  will  perform 
assigned  functions  to  assist  this  synthesis 
activity. 

The  verification  and  decision  activity  al¬ 
lows  tradeoff  of  alternative  approaches 
to  "how."  This  activity  is  conducted  in 
accordance  with  decision  criteria  set  by 
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higher-level  technical  requirements  for  such 
things  as  life-cycle  costs,  effectiveness,  reli¬ 
ability,  availability,  maintainability,  risk 
limits,  schedule,  etc.  It  is  repeated  at  each 
level  of  development.  The  verification  and 
decision  activity  is  assisted  by  develop¬ 
mental  testers  during  the  later  Program 
Definition  and  Risk  Reduction  Phase  when 
competitive  testing  between  alternative  ap¬ 
proaches  is  performed. 

The  final  activity  is  a  description  of  system 
elements.  Developing  as  the  result  of  previ¬ 
ous  activities  and  as  the  final  system  design 
is  determined,  this  activity  takes  form  when 
specifications  are  verified  through  testing 
and  when  reviewed  in  the  Physical  Con¬ 
figuration  and  Functional  Configuration 
Audits.  During  Phase  II,  operational  testers 
assist  in  this  activity.  They  conduct  opera¬ 
tional  testing  of  the  test  items/systems  to 
help  determine  the  personnel,  equipment, 
facilities,  software  and  technical  data  re¬ 
quirements  of  the  new  system  when  used 
by  typical  military  personnel.  Figure  2-3, 
Systems  Engineering  Process  and  Test  and 
Evaluation,  depicts  the  activities  and  their 
interactions. 

2.3.2  Technical  Management 
Planning 

The  technical  management  planning  incor¬ 
porates  top-level  management  planning  for 
the  integration  of  all  system  design  activi¬ 
ties.  Its  purpose  is  to  develop  the  organiza¬ 
tional  mechanisms  for  direction  and  con¬ 
trol,  and  identify  personnel  for  the  attain¬ 
ment  of  cost,  performance  and  schedule 
objectives.  Planning  defines  and  describes 
the  type  and  degree  of  system  engineering 
management,  the  systems  engineering  pro¬ 
cess,  and  the  integration  of  related  engi¬ 
neering  programs.  The  design  evolution 
process  forms  the  basis  for  comprehensive 
test  and  evaluation  planning. 

The  TEMP  must  be  consistent  with  tech¬ 


nical  management  planning.  The  testing 
program  outlined  in  the  TEMP  must  pro¬ 
vide  the  technical  performance  measure¬ 
ments  data  required  for  all  design  decision 
points,  audits  and  reviews  that  are  a  part  of 
the  system's  engineering  process.  The  con¬ 
figuration  management  process  controls 
the  baseline  for  the  test  programs  and  in¬ 
corporates  design  modifications  to  the 
baseline  determined  to  be  necessary  by 
T&E. 

The  TEMP  and  technical  management  plan¬ 
ning  must  be  traceable  to  each  other.  The 
system  description  in  the  TEMP  must  be 
traceable  to  systems  engineering  documen¬ 
tation  such  as  the  FFBDs,  the  RASs,  and  the 
Test  Requirements  Sheets  (TRSs).  Key  func¬ 
tions  and  interfaces  of  the  system  with 
other  systems  must  be  described  and  corre¬ 
lated  with  the  systems  engineering  docu¬ 
mentation  and  the  system  specification. 
Technical  thresholds  and  objectives  include 
specific  performance  requirements  that  be¬ 
come  test  planning  limits.  They  must  be 
traceable  through  the  planned  systems  en¬ 
gineering  documentation  and  can  be  corre¬ 
lated  to  the  content  of  the  Technical  Perfor¬ 
mance  Measurement  (TPM)  Program.  For 
example,  failure  criteria  for  reliability 
thresholds  during  OT&E  testing  must  be 
delineated  and  agreed  upon  by  the  pro¬ 
gram  manager  and  the  operational  test  di¬ 
rector  and  reflected  in  the  TEMP. 

2.3.3  Technical  Performance 
Measurement 

TPM  identifies  critical  technical  parameters 
that  are  at  a  higher  level  of  risk  during 
design.  It  tracks  evaluation  and  test  data, 
makes  predictions  about  whether  the  pa¬ 
rameter  can  achieve  final  technical  success 
within  the  allocated  resources,  and  assists 
in  managing  the  technical  program. 

The  TPM  Program  is  an  integral  part  of 
the  T&E  program.  The  TPM  is  defined  as 
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Figure  2-3.  Systems  Engineering  and  Test  and  Evaluation 


product  design  assessment  and  forms  the 
backbone  of  the  development  testing  pro¬ 
gram.  It  estimates,  through  engineering 
analyses  and  tests,  the  values  of  essential 
performance  parameters  of  the  current  pro¬ 
gram  design.  It  serves  as  a  major  input  in 
the  continuous  overall  evaluation  of  opera¬ 
tional  effectiveness  and  suitability.  Design 
reviews  are  conducted  to  measure  the  sys¬ 
tems  engineering  progress.  For  more  in¬ 
formation,  see  Chapter  8.  Figure  2-4  de¬ 
picts  the  technical  reviews  that  usually 
take  place  during  the  systems  engineer¬ 
ing  process  and  the  related  specification 
documents. 

2.3.4  System  Baselining  and  T&E 

The  systems  engineering  process  estab¬ 
lishes  phase  baselines  throughout  the  ac¬ 
quisition  cycle.  These  baselines  (functional, 
allocated,  product)  can  be  modified  with 
the  results  of  engineering  and  testing.  The 
testing  used  to  prove  the  technical  baselines 
is  rarely  the  same  as  the  operational  testing 
of  requirements. 

Related  to  the  baseline  is  the  process  of 
configuration  management.  Configuration 
management  benefits  the  test  and  evalua¬ 
tion  community  in  two  ways.  Through  con¬ 
figuration  management,  the  baseline  to  be 
used  for  testing  is  determined.  Also, 
changes  that  occur  to  the  baseline  as  a 
result  of  testing  and  design  reviews  are 
incorporated  into  the  test  article  before  the 
new  phase  of  testing  (to  prevent  retest  of  a 
bad  design). 

2.4  DEFINITIONS 

Test  and  evaluation  is  the  deliberate  and 
rational  generation  of  performance  data, 
which  concerns  the  nature  of  the  emerging 
system  and  the  transformation  of  data  into 
information  useful  to  the  technical  and 
managerial  personnel  controlling  its  devel¬ 


opment.  In  the  broad  sense,  T&E  may  be 
defined  as  all  physical  testing,  modeling, 
simulation,  experimentation  and  related 
analyses  performed  during  research,  de¬ 
velopment,  introduction  and  employment 
of  a  weapon  system  or  subsystem.  The  Glos¬ 
sary:  Defense  Acquisition  Acronyms  and  Terms, 
produced  by  the  Defense  Systems  Manage¬ 
ment  College  defines  "Test"  and  "Test  and 
Evaluation"  as  follows: 

A  "test"  is  any  program  or  procedure 
which  is  designed  to  obtain,  verify,  or 
provide  data  for  the  evaluation  of:  re¬ 
search  and  development  (other  than 
laboratory  experiments);  progress  in 
accomplishing  development  objec¬ 
tives;  or  performance  and  operational 
capability  of  systems,  subsystems, 
components,  and  equipment  items. 

"Test  and  Evaluation"  is  the  process  by 
which  a  system  or  components  pro¬ 
vide  information  regarding  risk  and 
risk  mitigation  and  empirical  data  to 
validate  models  and  simulations.  T&E 
permit,  as  assessment  of  the  attain¬ 
ment  of  technical  performance,  speci¬ 
fications  and  system  maturity  to  de¬ 
termine  whether  systems  are  opera¬ 
tionally  effective,  suitable  and  surviv- 
able  for  intended  use.  There  are  two 
types  of  T&E  —  Developmental 
(DT&E)  and  Operational  (OT&E). 

2.5  THE  DoD  TEST  AND 
EVALUATION  PROCESS 

The  Test  and  Evaluation  Process  (Figure  2- 
5)  is  an  iterative  five  step  process  that  pro¬ 
vides  answers  to  critical  T&E  questions  for 
decision  makers  at  various  times  during  a 
system  acquisition.  The  T&E  process  be¬ 
gins  during  the  formative  stages  of  the 
program  with  the  T&E  Coordination  Func¬ 
tion,  in  which  the  information  needs  of  the 
various  decision  makers  are  formulated  in 
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Figure  2-4.  Design  Reviews 


conjunction  with  the  development  of  the 
program  requirements,  acquisition  strat¬ 
egy  and  analysis  of  alternatives. 

Given  certain  foundation  documentation. 
Step  1  is  the  identification  of  T&E  informa¬ 
tion  required  by  the  decision-maker.  The 
required  information  usually  centers  on 
the  current  system  under  test  which  may  be 
in  the  form  of  concepts,  prototypes,  engi¬ 
neering  development  models,  or  produc¬ 
tion  representative /production  systems, 
depending  on  the  acquisition  phase.  The 
required  information  consists  of  perfor¬ 
mance  evaluations  of  effectiveness  and  suit¬ 
ability,  providing  insights  into  how  well 
the  system  meets  the  use's  needs  at  a  point 
in  time. 

Step  2  is  the  pre-test  analysis  of  the  evalua¬ 
tion  objectives  from  Step  1  to  determine  the 
types  and  quantities  of  data  needed,  the 
results  expected  or  anticipated  from  the 
tests,  and  the  analytical  tools  needed  to 
conduct  the  tests  and  evaluations,  the  use 
of  validated  models  and  simulation  sys¬ 
tems  during  pre-test  analysis  can  aid  in 
determining:  how  to  design  test  scenarios; 
how  to  set  up  the  test  environment;  how  to 
properly  instrument  the  test;  how  to  staff 
and  control  test  resources;  how  best  to  se¬ 
quence  the  test  trials;  and  how  to  estimate 
outcomes. 

Step  3,  test  activity  and  data  management, 
is  the  actual  test  activity  planning,  tests  are 
conducted,  and  data  management  for  data 
requirements  are  identified  in  Step  2.  T&E 
managers  determine  what  valid  data  exists 
in  historical  files  that  can  be  applied  and 
what  new  data  must  be  developed  through 
testing.  The  necessary  tests  are  planned 
and  executed  to  accumulate  sufficient  data 
to  support  analysis.  Data  is  screened  for 
completeness,  accuracy,  and  validity  be¬ 
fore  being  used  for  Step  4. 


Step  4,  post  test  synthesis  and  evaluation,  is 
the  comparison  of  the  measured  outcomes 
(test  data)  from  Step  3  with  the  expected 
outcomes  from  Step  2,  tempered  with  tech¬ 
nical  and  operational  judgment.  This  is 
where  data  is  S3mthesized  into  informa¬ 
tion.  When  the  measured  outcomes  differ 
from  the  expected  outcomes,  the  test  condi¬ 
tions  and  procedures  must  be  reexamined 
to  determine  if  the  performance  deviations 
are  real  or  were  the  result  of  test  conditions, 
such  as  lack  of  fidelity  in  computer  simula¬ 
tion,  insufficient  or  incorrect  test  support 
assets,  instrumentation  error,  or  faulty  test 
processes.  The  assumptions  of  tactics,  op¬ 
erational  environment,  systems  perfor¬ 
mance  parameters,  and  logistic  support 
must  have  been  carefully  chosen,  fully  de¬ 
scribed,  and  documented  prior  to  test. 
Modeling  and  simulation  may  normally  be 
used  during  the  data  analysis  to  extend  the 
evaluation  of  performance  effectiveness  and 
suitability. 

Step  5  is  when  the  decision  maker  weighs 
the  T&E  information  against  other  pro¬ 
grammatic  information  to  decide  a  proper 
course  of  action.  This  process  may  identify 
additional  requirements  for  test  data  and 
iterate  the  DoD  T&E  process  again. 

2.6  SUMMARY 

Test  and  evaluation  is  an  engineering  tool 
used  to  identify  technical  risk  throughout 
the  defense  system  acquisition  cycle.  This 
iterative  cycle  consists  of  acquisition  phases 
separated  by  discrete  milestones.  The  DoD 
T&E  process  consists  of  developmental  and 
operational  testing  that  is  used  to  support 
engineering  design  and  programmatic  re¬ 
views.  This  T&E  process  forms  an  impor¬ 
tant  part  of  the  system  engineering  process 
used  by  system  developers  and  aids  in  the 
milestone  decision  process  used  by  senior 
decision  authorities  in  DoD. 
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Figure  2-5.  DoD  Test  and  Evaluation  Process 


T&E  POLICY  STRUCTURE 
AND  OVERSIGHT  MECHANISM 


3.1  INTRODUCTION 

This  chapter  provides  an  overview  of  the 
policy  and  organizations  that  govern  the 
conduct  of  test  and  evaluation  (T&E)  ac¬ 
tivities  within  the  Department  of  Defense 
(DoD)  and  discusses  congressional  legisla¬ 
tion  and  activities  for  compliance  by  DoD. 
It  outlines  the  responsibilities  of  DoD  test 
organizations  at  the  Office  of  the  Secretary 
of  Defense  (OSD)  and  Service  levels,  and 
describes  related  T&E  policy. 

3.2  THE  CONGRESS 

The  Congress  has  shown  a  long-standing 
interest  in  influencing  the  DoD  acquisition 
process.  During  the  early  1970s,  in  response 
to  urging  by  the  Congress  and  recommen¬ 
dations  by  a  Presidential  Blue  Ribbon  Panel 
on  Defense  Management,  the  Deputy  Sec¬ 
retary  of  Defense,  David  Packard,  promul¬ 
gated  a  package  of  policy  initiatives  that 
established  the  Defense  Systems  Acquisi¬ 
tion  Review  Council  (DS ARC) .  The  DSARC 
was  organized  to  resolve  acquisition  is¬ 
sues,  whenever  possible,  and  to  provide 
recommendations  to  the  Secretary  of  De¬ 
fense  (SECDEF)  on  the  acquisition  of  major 
weapon  systems.  Also,  as  a  result  of  the 
Congressional  Directives,  the  Army  and 
Air  Force  established  independent  opera¬ 
tional  test  agencies.  The  Navy  Operational 
Test  and  Evaluation  Force  was  established 
in  the  late  1960s.  In  1983,  similar  concerns 
led  the  Congress  to  direct  the  establish¬ 
ment  of  the  independent  Office  of  the  Di¬ 
rector,  Operational  Test  and  Evaluation 


(EXDT&E),  within  OSD.  In  1985  a  report 
released  by  another  President’s  Blue  Rib¬ 
bon  Commission  on  Defense  Management, 
this  time  chaired  by  David  Packard,  made 
significant  recommendations  on  the  man¬ 
agement  and  oversight  of  DoD's  acquisi¬ 
tion  process,  specifically,  T&E.  All  the 
Commission's  recommendations  have  not 
been  implemented,  and  the  full  impact  of 
these  recommendations  is  not  yet  realized. 
In  fiscal  year  (FY)87  the  Defense  Authori¬ 
zation  Act  required  live  fire  testing  of 
weapon  systems  before  the  Production 
Phase  begins.  The  earmarking  of  authori¬ 
zations  and  appropriations  for  DoD  fund¬ 
ing,  and  acquisition  reform  legislation  con¬ 
tinues  to  indicate  the  will  of  the  Congress 
for  DoD  implementation. 

Congress  requires  DoD  to  provide  the  fol¬ 
lowing  reports  on  test  and  evaluation: 

•  Selected  Acquisition  Report  (SAR). 
Within  the  cost,  schedule  and  performance 
data  in  the  report,  SAR  describes  Acquisi¬ 
tion  Category  (ACAT)  I  system  character¬ 
istics  required  and  outlines  significant 
progress  and  problems  encountered.  It  lists 
tests  completed  and  issues  identified  dur¬ 
ing  testing. 

•  Annual  System  Operational  Test  Re¬ 
port.  This  report  is  provided  by  the  DOT &E 
to  the  SECDEF  and  the  committees  on 
Armed  Services,  National  Security,  and 
Appropriations.  The  report  provides  a 
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narrative  and  resource  summary  of  all 
Operational  Test  and  Evaluation  (OT&E) 
and  related  issues,  activities,  and  assess¬ 
ments.  When  oversight  of  live  fire  testing 
was  moved  to  DOT&E,  this  issue  was 
added  to  the  report. 

•  Beyond  Low  Rate  Initial  Production 
(BLRIP)  Report.  Before  proceeding  BLRIP 
for  each  major  system  acquisition  pro¬ 
gram,  DOT&E  must  report  to  the  SECDEF 
and  the  Congress.  This  report  addresses 
the  adequacy  of  OT&E  and  whether  the 
T&E  results  confirm  that  the  tested  item 
or  component  is  effective  and  suitable  for 
combat.  When  oversight  of  live  fire  test¬ 
ing  was  moved  to  the  DOT&E,  the  Live 
Fire  Test  Report  was  added  to  the  BLRIP 
report  content. 

•  Foreign  Comparative  Test  Report. 
The  Director,  Test,  Systems  Engineering, 
and  Evaluation  (DTSEE)  should  notify 
the  Congress  a  minimum  of  30  days  prior 
to  the  commitment  of  funds  for  initiation 
of  new  FCT  evaluations. 

3.3  OSD  OVERSIGHT  STRUCTURE 

The  DoD  organization  for  the  oversight 
of  T&E  is  illustrated  in  Figure  3-1.  In 
OSD,  T&E  oversight  is  performed  by  two 
primary  offices:  the  DTSEE  and  DOT&E. 
The  management  of  major  defense  acqui¬ 
sition  programs  in  OSD  is  performed  by 
the  Defense  Acquisition  Executive  (DAE), 
who  uses  the  Defense  Acquisition  Board 
(DAB)  and  Overarching  Integrated  Prod¬ 
uct  Teams  (OIPT)  to  process  information 
for  decisions.  The  designated  DAE  is  the 
Under  Secretary  of  Defense  for  Acquisi¬ 
tion  and  Technology  (USD(A&T))  who 
uses  the  DAB  and  its  OIPTs  to  provide 
the  senior-level  decision  process  for  the 
acquisition  of  weapon  systems. 


3.3.1  Defense  Acquisition 
Executive  (DAE) 

The  DAE  position,  established  in  Sep¬ 
tember  1986,  is  held  by  the  USD(A&T). 
The  responsibilities  include,  establishing 
policies  for  acquisition  (including  pro¬ 
curement,  research  and  development,  lo¬ 
gistics,  development  testing,  and  con¬ 
tracts  administration)  for  all  elements  of 
DoD.  His  charter  includes  the  authority 
over  the  Service  and  defense  agencies  on 
policy,  procedure  and  execution  of  the 
acquisition  process. 

3.3.2  Defense  Acquisition  Board 
(DAB) 

The  DAB  is  the  primary  forum  used  by 
OSD  to  provide  advice,  assistance  and 
recommendations,  and  to  resolve  issues 
regarding  all  operating  and  policy  as¬ 
pects  of  the  DoD  acquisition  system.  The 
DAB  is  the  senior  management  acquisi¬ 
tion  board  chaired  by  the  DAE  and  Vice 
Charman  is  the  Vice  Chairman  of  the 
Joint  Chiefs  of  Staff.  The  DAB  is  com¬ 
posed  of  the  department's  senior  acquisi¬ 
tion  officials,  including  the  EXDT&E.  The 
DAB  conducts  business  through  OIPTs 
and  provides  decisions  on  AC  AT  ID  pro¬ 
grams  (DoD  5000.2-R). 

3.3.3  Defense 
Resources  Board  (DRB) 

The  DRB  was  established  by  the  SECDEF 
in  1979  to  advise  the  SECDEF  on  policy, 
planning,  program  and  budget  issues. 
The  DRB  is  chaired  by  the  Deputy  Secre¬ 
tary  of  Defense  and  is  responsible  for  the 
management  and  oversight  of  all  aspects 
of  the  DoD  planning,  programming  and 
budgeting  process.  It  oversees  the  bud¬ 
get  review  processes.  Therefore,  DRB  has 
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Figure  3-1.  DoD  Test  and  Evaluation  Organization 


a  major  impact  on  test  and  evaluation  re¬ 
sources. 

3.3.4  Director  Test,  Systems 
Engineering,  and  Evaluation 
(DTSEE) 

The  DTSEE  serves  as  the  principal  staff 
assistant  and  advisor  to  the  USD(A&T)  for 
T&E  matters.  The  DTSEE  works  for  the 
Principle  Deputy  USD(A&T)  and  has  au¬ 
thority  and  responsibility  for  all  Develop¬ 
ment  Test  and  Evaluation  (DT&E)  con¬ 
ducted  on  designated  major  defense  acqui¬ 
sition  programs.  The  DTSEE  is  illustrated 
in  Figure  3-1. 

3.3.4.1  Duties  of  the  DTSEE 

Within  the  acquisition  community,  the 
DTSEE: 

•  Serves  as  the  focal  point  for  coordina¬ 
tion  of  all  major  defense  acquisition  pro¬ 
gram  test  and  evaluation  master  plans 
(TEMPs).  Signs  for  approval  of  the  DT&E 
portion  of  TEMPs; 

•  Reviews  major  defense  acquisition  pro¬ 
gram  documentation  for  DT&E  implica¬ 
tions  and  resource  requirements  to  provide 
comments  to  the  USD(  A&T),  DAE  or  DAB; 

•  Observes  DT&E  to  ensure  adequacy  of 
testing  and  to  assess  test  results; 

•  Provides  a  technical  assessment  of 
DT&E  and  system  engineering  processes 
conducted  on  a  weapon  system; 

•  Provides  advice  and  makes  recom¬ 
mendations  to  the  SECDEF,  and  issues 
guidance  to  the  component  acquisition  ex¬ 
ecutives  with  respect  to  DT&E; 

•  Performs  the  administrative  process¬ 


ing  of  nominations  and  charters  for  Joint 
Development  Test  and  Evaluation  pro¬ 
grams; 

•  Provides  management  oversight  of  the 
Major  Range  and  Test  Fadhty  Base  (MRTFB) 
and  acquisition  of  threat  targets/ simula¬ 
tors; 

•  Administers  the  Foreign  Comparative 
Test  Program. 

3.3.4.2  DTSEE  and  Service  Reports 

During  the  testing  of  ACAT  I  and  desig¬ 
nated  weapon  systems,  the  DTSEE  and 
Services  interaction  includes  the  following 
reporting  requirements: 

•  A  TEMP  (either  prehminary  or  up¬ 
dated,  as  appropriate)  must  be  provided 
for  consideration  and  approval  before  each 
milestone  review,  starting  with  Milestone 
(MS)  I. 

•  An  End-of-Test  Phase  Report  must  be 
provided  to  the  DTSEE  and  EXDT&E  listing 
the  T&E  results,  conclusions  and  recom¬ 
mendations  prior  to  a  milestone  decision  or 
the  final  decision  to  proceed  BLRIP. 

3.3.5  Director,  Operational  Test 
and  Evaluation  (DOT&E) 

As  illustrated  in  Figure  3-1,  the  director 
reports  directly  to  the  SECDEF  and  has 
special  reporting  requirements  to  the  Con¬ 
gress.  The  DOT&E's  responsibility  to  the 
Congress  is  to  provide  an  unbiased  win¬ 
dow  of  insight  into  the  operational  effec¬ 
tiveness,  suitability,  and  survivability  of 
new  weapon  systems. 

3.3.5.1  Duties  and  Functions 
of  the  DOT&E 

The  specific  duties  of  DOT&E  are  outlined 
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in  DoD  Directive  5141.2  (Reference  13). 
The  functions  of  the  office  include: 

•  Obtaining  reports,  information,  advice 
and  assistance  as  necessary  to  carry  out 
assigned  functions  DOT&E  has  access  to  all 
records  and  data  in  DoD  on  acquisition 
programs); 

•  Signing  the  ACAT  I  and  oversight 
TEMPs  for  approval  of  OT&E  and  approv¬ 
ing  the  OT&E  funding  for  major  systems 
acquisition; 

•  Approving  operational  test  plans  on  all 
major  defense  acquisition  systems  and  des¬ 
ignated  oversigh  programs  prior  to  system 
starting  initial  operational  testing  (approval 
in  writing  required  before  operational  test¬ 
ing  may  begin)  oversight  extends  into  fol¬ 
low-on  OT&E; 

•  Providing  observers  during  prepara¬ 
tion  and  conduct  of  OT&E; 

•  Analyzing  results  of  OT&E  conducted 
for  each  major  or  designated  defense  acqui¬ 
sition  program  and  submitting  a  report  to 
the  SECDEF  and  the  Congress  on  the  ad¬ 
equacy  of  the  OT&E  performed; 

•  A  final  decision  to  proceed  with  a 
major  program  BLRIP  cannot  be  made  un¬ 
til  DOT&E  has  reported  (BLRIP  Report)  to 
the  SECDEF  and  to  congressional  Commit¬ 
tees  on  Armed  Services  and  Appropria¬ 
tions  on  the  adequacy  of  live  fire  and  opera¬ 
tional  T&E  and  whether  the  results  confirm 
the  system's  operational  effectiveness  and 
suitability; 

•  Provide  oversight  and  approval  of 
major  program  live  fire  testing. 

3.3.5.2  DOT&E  and  Service  Interactions 

For  DoD  and  IX)T&E-designated  acquisi¬ 
tion  programs,  the  Service  provides  the 


DOT&E  the  following: 

•  A  draft  copy  of  the  Operational  Test 
Plan  concept  for  review; 

•  Significant  Test  Plan  changes; 

•  The  final  Service  lOT&E  report  which 
must  be  submitted  to  DOT&E  before  the 
DAB  Milestone  III  review; 

•  The  live  fire  T&E  plan  for  approval  and 
the  Service  live  fire  test  report  for  review. 

3.4  SERVICE  T&E  MANAGEMENT 
STRUCTURES 

3.4.1  Army  T&E  Organizational 
Relationship 

The  Army  management  structure  for  T&E 
is  illustrated  in  Figure  3-2. 

3.4.1.1  Army  Acquisition 
Executive 

The  Under  Secretary  of  the  Army  is  the 
Army  Acquisition  Executive  (AAE).  The 
AAE  is  responsible  for  all  acquisition  T&E 
(operational  and  developmental  tests)  plan¬ 
ning,  programming,  budgeting,  and  devel¬ 
opmental  testing  policy  and  oversight.  The 
AAE  performs  these  duties  with  the  assis¬ 
tance  of  the  Assistant  Secretary  of  the  Army, 
Research,  Development,  and  Acquisition 
(ASA/RDA).  As  illustrated  in  Figure  3-2, 
the  ASA/RDA  is  organized  to  provide  tech¬ 
nical  assessments  and  program  evaluations. 
The  ASA/RDA  resolves  acquisition  issues 
whenever  possible  and  recommends  ac¬ 
quisition  of  weapon  systems  to  the  AAE. 
The  Deputy  Under  Secretary  of  the  Army 
for  Operations  Research  (DUSA(OR))  is 
chartered  to  supervise  all  Army  T&E  policy 
and  has  oversight  for  all  Army  T&E.  This 
oversight  is  provided  by  the  Test  and  Evalu¬ 
ation  Management  Agency  within  the  Of¬ 
fice  of  the  Chief  of  Staff. 
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Figure  3-2.  Army  T&E  Organization 


3.4.1.2  Army  Technical  Testers 

The  U.S.  Army  Materiel  Command  (AMC) 
is  responsible  for  the  management  of  DT&E. 
The  Test  and  Evaluation  Command 
(TECOM)  has  the  primary  responsibility 
for  conducting  technical  tests  for  the  Army. 

The  TECOM  is  responsible  for: 

•  Planning,  executing  and  reporting  the 
results  of  technical  tests.  Technical  tests 
include  development  tests,  technical  feasi¬ 
bility  tests,  production  qualification  tests, 
joint  tests  and  contractor/foreign  tests; 

•  Providing  test  facilities  and  technical 
expertise  in  support  of  the  T&E  life  cycle; 

•  Maintaining  the  Army's  Major  Range 
and  Test  Facility  Base; 

•  Maintaining  Army's  facilities  which 
make  up  part  of  the  MRTFB; 

•  Researching,  developing  and  acquir¬ 
ing  instrumentation  and  developing  new 
and  improved  test  methodology; 

•  Providing  safety  confirmations. 

3.4.1.3  Army  Operational  Test 
and  Evaluation  Command 

•  The  Army  Operational  Test  and  Evalu¬ 
ation  Command  (OPTEC)  is  responsible 
for  the  management  of  operational  testing 
and  evaluation  as  well  as  the  management 
of  joint  user  testing.  The  OPTEC  is  an  inde¬ 
pendent  agency  reporting  directly  to  the 
Army  Vice  Chief  of  Staff. 

•  The  OPTEC  combines  the  evaluation 
function  performed  for  both  DT&E  and 
OT&E  in  the  Operational  Evaluation  Com¬ 
mand  (OEC)  (projected  to  become  the  Army 
Evaluation  Command)  and  the  operational 


testing  function  performed  by  the  Test  and 
Experimentation  Command  (TEXCOM). 

•  The  U.S  Army  Forces  Command 
(FORSCOM)  supports  testing  by  provid¬ 
ing  user  troops  and  facilities  as  needed. 

3.4.2  Navy  T&E  Organizational 
Relationship 

The  organizational  structure  for  T&E  in  the 
Navy  is  illustrated  in  Figure  3-3.  Within  the 
Navy  Secretariat,  the  Secretary  of  the  Navy 
has  assigned  general  and  specific  research, 
development,  test  and  evaluation  (RDT&E) 
responsibilities  to  the  Assistant  Secretary  of 
the  Navy  (Research,  Development  and  Ac¬ 
quisition)  and  to  the  Chief  of  Naval  Opera¬ 
tions  (CNO).  The  CNO  has  responsibility 
for  ensuring  the  adequacy  of  the  Navy' s 
overall  test  and  evaluation  program.  The 
T&E  policy  and  guidance  are  exercised 
through  the  Directorate  of  Navy;  T&E 
and  Technology  Requirements  (N-91). 
Staff  support  is  provided  by  the  Test  and 
Evaluation  Division  (N-91 2)  which  has 
cognizance  over  planning,  conducting 
and  reporting  all  T&E  associated  with 
development  of  systems. 

3.4.2.2  Navy  DT&E  Organizations 

The  Navy's  senior  systems  development  au¬ 
thority  is  divided  among  the  commanders  of 
the  system  commands  with  Naval  Air  Sys¬ 
tems  Command  (NAVAJDR)  developing  and 
performing  DT&E  on  aircraft  and  their  es¬ 
sential  weapon  systems;  Naval  Sea  Systems 
Command  (NAVSEA)  developing  and  per¬ 
forming  DT &E  on  ships,  submarine  and  their 
associated  weapon  systems  and  Space  and 
Naval  Systems  Warfare  Command 
(SPAWAR)  developing  and  performing 
DT&E  on  all  other  systems.  System  acquisi¬ 
tion  is  controlled  by  a  chartered  program 
manager  or  by  the  commander  of  a  systems 
command.  In  both  cases,  the  designated  de- 
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Figure  3-3.  Navy  T&E  Organization 


veloping  agency  is  responsible  for  DT&E 
and  for  the  coordination  of  all  test  and 
evaluation  planning  in  the  TEMP.  Devel¬ 
oping  Agencies  (DAs)  are  responsible  for: 

•  Developing  test  issues  based  on  the 
thresholds  established  by  the  user  in  the 
Operational  Requirements  Document; 

•  Identifpng  the  testing  facilities  and 
resources  required  to  conduct  the  DT&E; 

•  Developing  the  DT&E  test  reports  and 
quick-look  reports. 

3.4.2.3  Navy  Operational  Test 
and  Evaluation  Force 

The  Commander,  Operational  Test  and 
Evaluation  Force  (COMOPTEVFOR),  com¬ 
mands  the  Navy 's  independent  operational 
test  and  evaluation  activity  and  reports 
directly  to  the  CNO.  The  functions  of  the 
COMOPTEVFOR  include: 

•  Establishing  early  liaison  with  the  DA 
to  ensure  an  understanding  of  the  test  re¬ 
quirements  and  plans; 

•  Reviewing  acquisition  program  docu¬ 
mentation  to  ensure  that  documents  are 
adequate  to  support  a  meaningful  T&E 
program; 

•  Planning  and  conducting  realistic 
OT&E; 

•  Developing  tactics  and  procedures  for 
the  employment  of  systems  that  undergo 
OT&E  (as  directed  by  the  CNO); 

•  Providing  recommendations  to  the 
CNO  for  the  development  of  new  capabili¬ 
ties  or  the  upgrade  of  ranges; 

•  Also  reporting  directly  to  the  CNO,  the 
President  of  the  Board  of  Inspection  and 


Survey  (PRESINSURV)  is  responsible  for 
conducting  acceptance  trials  of  new  ships 
and  aircraft  acquisitions  and  is  the  primary 
Navy  authority  for  production  acceptance 
T&E  of  these  systems; 

•  Conducting  OT&E  on  aviation  sys¬ 
tems  in  conjunction  with  Marine  Corps 
Operational  Test  and  Evaluation  Activity 
(MCOTEA). 

3.4.3  Air  Force  Organizational 
Relationships 

3.4.3.1  Air  Force  Acquisition 
Executive 

The  Assistant  Secretary  of  the  Air  Force  for 
Acquisition  (ASAF/AQ)  is  the  senior-level 
authority  for  research,  development  and 
acquisition  within  the  Air  Force.  As  illus¬ 
trated  in  Figure  3-4,  the  ASAF/AQ  is  an 
advisor  to  the  Secretary  of  the  Air  Force 
and  interfaces  directly  with  the  DT&E  and 
DOT&E.  The  ASAF/AQ  receives  DT&E 
and  OT&E  results  as  a  part  of  the  acquisi¬ 
tion  decision  process.  Within  the  ASAF/ 
AQ  structure,  there  is  a  military  deputy 
(acquisition)  who  is  the  Air  Force  primary 
staff  officer  with  responsibility  for  RD&A. 
This  staff  officer  is  the  chief  advocate  of  Air 
Force  acquisition  programs  and  develops 
the  RDT&E  budget.  Air  Force  policy  and 
oversight  for  T&E  is  provided  by  a  staff 
element  under  the  Chief  of  Staff,  Test  and 
Evaluation  (AF/TE).  They  process  test 
documentation  for  DT&E  and  OT&E  and 
manage  the  review  of  the  TEMP. 

3.4.3.2  Air  Force  DT&E 
Organization 

The  Air  Force  Materiel  Command  (AFMC) 
is  the  primary  DT&E  and  acquisition  man¬ 
ager.  The  AFMC  performs  all  levels  of  re¬ 
search;  develops  weapon  systems,  support 
systems  and  equipment;  and  conducts  all 
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Figure  3-4.  Air  Force  T&E  Organization 


DT&E.  The  acquisition  program  managers 
are  under  the  Commander,  AFMC.  Within 
the  AFMC,  there  are  major  product  divi¬ 
sions,  test  centers  and  laboratories  as  well 
as  missile,  aircraft  and  munitions  test 
ranges. 

Once  the  weapon  system  is  fielded,  AFMC 
retains  management  responsibility  for  de¬ 
veloping  and  testing  system  improvements, 
enhancements  or  upgrades. 

3.4.33  Air  Force  OT&E  Organization 

The  AF/TE  is  responsible  for  supporting 
and  coordinating  the  OT&E  activities  of  the 
Air  Force  Operational  Test  and  Evaluation 
Center  (AFOTEC). 

The  Commander,  AFOTEC,  is  responsible 
to  the  Secretary  of  the  Air  Force  and  the 
Chief  of  Staff  for  the  independent  test  and 
evaluation  of  all  major  and  nonmajor  sys¬ 
tems  acquisitions.  The  Commander  is  sup¬ 
ported  by  the  operational  commands  and 
others  in  planning  and  conducting  OT&E. 

The  AFOTEC  reviews  operational  require¬ 
ments,  employment  concepts,  tactics,  main¬ 
tenance  concepts,  training  requirements 
before  conducting  OT&E.  The  operational 
commands  provide  operational  concepts, 
personnel  and  resources  to  assist  AFOTEC 
in  performing  OT&E. 

3.4.4  Marine  Corps  Organizational 
Relationship 

3.4.4.1  Marine  Corps  Acquisition 
Executive 

The  Deputy  Chief  of  Staff  for  Research  and 
Development  (DCS/R&D),  Headquarters 
Marine  Corps,  directs  the  total  Marine 
Corps  RDT&E  effort  to  support  the  acqui¬ 
sition  of  new  systems.  The  DCS/R&D's 
position  within  the  General  Staff  is  analo¬ 
gous  to  that  of  the  Director,  T&E,  Tech/N- 


91  in  the  Navy  structure.  The  DCS/R&D 
also  reports  directly  to  the  Assistant  Secre¬ 
tary  of  the  Navy/Research,  Engineering 
and  Science  (ASN/RE&S)  in  the  Navy  Sec¬ 
retariat.  Figure  3-3,  illustrates  the  Marine 
Corps  organization  for  T&E  management. 

3.4.4.2  Marine  Corps  DT&E 
Organizations 

The  Commanding  General,  Marine  Corps 
Systems  Command  (CG  MCSC),  is  the 
Marine  Corps  materiel  developing  agent 
and  directly  interfaces  with  the  Navy  Sys¬ 
tems  Commands.  The  CG  MCSC  imple¬ 
ments  policies,  procedures  and  require¬ 
ments  for  DT&E  of  all  systems  acquired  by 
the  Marine  Corps.  The  Marine  Corps  also 
uses  DT&E  and  OT&E  performed  by  other 
Services,  which  may  develop  systems  of 
interest  to  the  Corps. 

3.4.4.3  Marine  Corps  Operational 

Test  and  Evaluation  Agency  (MCOTEA) 

The  MCOTEA  is  the  independent  OT&E  ac¬ 
tivity  maintained  by  the  Marine  Corps.  Its 
function  is  analogous  to  that  performed 
by  Operational  Test  and  Evaluation  Force 
(Navy)  (OPTEVFOR)  in  the  Navy.  The 
CG  MCSC  provides  direct  assistance  to 
MCOTEA  in  the  planning,  conduct  and 
reporting  of  OT&E.  The  Fleet  Marine 
Force  performs  troop  test  and  evaluation 
of  materiel  development  in  an  operational 
environment. 

3.5  THE  T&E  EXECUTIVE  AGENT 
STRUCTURE 

In  1993  the  USD(A&T)  approved  a  T&E 
Executive  Agent  structure  to  provide  the 
Services  with  more  corporate  responsibil¬ 
ity  for  the  management  and  policies  that 
influence  the  availability  of  test  resources 
for  the  evaluation  of  DoD  systems  in  acqui¬ 
sition  (Figure  3-5).  The  DTSEE  has  func- 
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Figure  3-5.  T&E  Executive  Agent  Structure 
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tional  responsibility  for  the  execution  of 
the  processes  necessary  to  assure  the  T&E 
Executive  Agent  structure  functions  effec¬ 
tively.  The  DTSEE  also  participates  in  the 
Operational  Test  and  Evaluation  Coordi¬ 
nating  Committee,  chaired  by  the  Director, 
Operational  Test  and  Evaluation.  This  com¬ 
mittee  manages  the  OT&E  Resources  En¬ 
hancement  Project  and  the  DTSEE  draws 
input  to  the  T&E  Executive  Agent  structure 
for  coordination  of  all  T&E  resource  re¬ 
quirements. 

The  Board  of  Directors  (BoD)  (Service  Vice 
Chiefs)  is  assisted  by  an  Executive  Secre¬ 
tariat  consisting  of  the  Army  DUSA(OR), 
the  Navy  N-91,  and  the  USAF  AF/TE.  The 
Board  of  Directors  provides  guidance  and 
decisions  on  policy  and  resource  allocation 
to  their  subordinate  element,  the  Board  of 
Operating  Directors  (TECOM  CG,  NAVAIR 
5.0,  and  AFMC  DO).  The  BoD  also  provides 
program  review  and  advocacy  support  of 
the  T&E  infrastructure  to  OSD  and  Con¬ 
gress. 

The  Board  of  Operating  Directors  (BoOD) 
is  supported  by  a  Secretariat  and  the  De¬ 
fense  Test  and  Training  Steering  Group 
(DTTSG).  The  DTTSG  manages  the  T&E 
Resources  Committee  (TERC),  the  Train¬ 
ing  Instrumentation  Resource  Investment 
Committee  (TIRIC),  and  the  CROSSBOW 
Committee.  The  DTTSG  is  instrumental  in 
achieving  efficient  acquisition  and  integra¬ 
tion  of  all  training  and  associated  test  range 
instrumentation  and  the  development  of 
acquisition  policy  for  embedded  weapon 
system  training  and  testing  capabilities. 
The  TERC  supports  the  DTTSG  in  oversee¬ 
ing  infrastructure  requirements  develop¬ 
ment  from  a  T&E  community  perspective, 
both  development  testing  and  operational 
testing,  and  manages  OSD  funding  the  ex¬ 
ecution  of  the  Central  T&E  Investment  Pro¬ 
gram  (CTEIP).  The  TIRIC  is  chartered  to 
ensure  the  efficient  acquisition  of  common 


and  interoperable  range  instrumentation 
systems.  The  CROSSBOW  Committee  pro¬ 
vides  technical  and  management  oversight 
of  the  Services'  development  and  acquisi¬ 
tion  programs  for  threat  and  threat  related 
hardware  simulators,  emitters,  software 
simulations,  hybrid  representations,  and 
surrogates. 

The  operating  arm  of  the  BoOD  is  the 
Joint  Program  Office  (JPO)  for  Test  and 
Evaluation  which  interfaces  with  the  Range 
Commander's  Council  and  the  T&E  Reli¬ 
ance  and  Investment  Board  (TERIB).  The 
JPO  manages  the  T&E  resource 
prioritization  through  the  CTEIP,  the  Test 
Resources  Master  Plan,  and  the  Test  Invest¬ 
ment  Strategy.  It  conducts  reviews  of  the 
MRTFB  assets,  manages  the  T&E  Corpo¬ 
rate  Information  system  and  conducts  other 
studies  as  directed  by  the  BoD/BoOD.  The 
Range  Commander's  Council  shares  its 
insights  and  products  with  various  Service 
and  DoD  oversight  boards  and  commit¬ 
tees,  and  stands  as  an  expert  consultant 
body  to  these  organizations.  The  TERIB 
provides  technical  leadership,  vision,  over¬ 
sight,  and  review  for  all  Service  T&E  in¬ 
vestment  planning  activities  to  foster  de¬ 
velopment  of  joint  investment  initiatives, 
to  ensure  the  development  and  sustain¬ 
ment  of  an  effective  and  efficient  defense 
T&E  capability,  to  prevent  unwarranted 
duplication  of  DoD  T&E  capabilities,  and 
to  optimize  the  Services'  investments  in 
T&E  capabilities. 

3.6  SUMMARY 

An  increased  emphasis  on  test  and  evalua¬ 
tion  has  placed  greater  demands  on  the 
OSD  and  DoD  components  to  carefully 
structure  organizations  and  resources  to 
ensure  maximum  effectiveness.  Renewed 
interest  by  Congress  in  testing  as  a  way  of 
assessing  systems  utility  and  effectiveness, 
the  report  by  the  President's  Blue  Ribbon 
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Panel  on  Acquisition  Management,  and 
acquisition  reform  initiatives  have  resulted 
in  major  reorganizations  within  the  Ser¬ 
vices.  These  policy  changes  and  reorgani¬ 


zations  will  be  ongoing  for  several  years  to 
improve  the  management  of  test  and  evalu¬ 
ation  resources  in  support  of  acquisition 
programs. 
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4 

PROGRAM  OFFICE  RESPONSIBILITIES 
FOR  TEST  AND  EVALUATION 


4.1  INTRODUCTION 

In  government  acquisition  programs,  there 
should  be  an  element  dedicated  to  manage¬ 
ment  of  test  and  evaluation  (T&E).  This 
element  would  have  the  overall  test  pro¬ 
gram  responsibility  for  all  phases  of  the 
acquisition  process.  T&E  expertise  may  be 
available  through  matrix  support  or  reside 
in  the  Program  Management  Office  (PMO) 
engineering  department  during  the 
program's  early  phases.  By  Engineering 
and  Manufacturing  Development  (EMD), 
the  PMO  should  have  a  dedicated  T&E 
manager.  In  the  PMO,  the  Deputy  for  T&E 
would  be  responsible  for  defining  the  scope 
and  concept  of  the  test  program,  establish¬ 
ing  the  overall  program  test  objectives  and 
managing  test  program  funds  and  coordi¬ 
nation.  The  Deputy  for  T&E  should  pro¬ 
vide  test  directors  (such  as  a  joint  test  direc¬ 
tor)  as  required,  and  coordinate  the  test 
resources,  facilities  and  their  support  re¬ 
quired  for  each  phase  of  testing.  In  addi¬ 
tion,  the  Deputy  for  T&E  or  a  staff  member, 
will  be  responsible  for  managing  the  Test 
and  Evaluation  Master  Plan  (TEMP)  and 
planning  and  managing  any  special  test 
programs  required  for  the  program.  The 
Deputy  for  T&E  will  also  review,  evaluate, 
approve  and  release  for  distribution  con¬ 
tractor-prepared  test  plans  and  reports  and 
review  and  coordinate  all  appropriate  gov¬ 
ernment  test  plans.  After  the  system  is  pro¬ 
duced,  the  Deputy  for  T&E  will  be  re¬ 
sponsible  for  supporting  production  ac¬ 
ceptance  testing  and  the  test  portions  of 
preplanned  product  improvements  (P^I) 


upgrades  or  enhancements  to  the  weapon 
system/acquisition.  If  the  program  is  large 
enough,  the  Deputy  for  T&E  will  be  re¬ 
sponsible  for  all  T&E  direction  and  guid¬ 
ance  for  that  program. 

4.2  RELATIONSHIP  TO 
THE  PROGRAM  MANAGER 

The  program  manager  (PM)  is  ultimately 
responsible  for  all  aspects  of  the  system 
development,  including  testing.  The 
Deputy  for  T&E  is  normally  authorized  by 
the  PM  to  conduct  all  duties  in  the  area  of 
test  and  evaluation.  The  input  of  the  Deputy 
for  T&E  to  the  contract,  engineering  speci¬ 
fications,  budget,  program  schedule,  etc.,  is 
essential  for  the  PM  to  manage  the  pro¬ 
gram  efficiently. 

4.3  EARLY  PROGRAM  STAGES 

In  the  early  stages  of  the  program,  the  T&E 
function  is  often  handled  by  matrix  sup¬ 
port  from  the  material  command.  Matrix 
T&E  support  or  the  Deputy  for  T&E  should 
be  responsible  for  development  of  the  test 
sections  of  the  Request  for  Proposal  (RFP). 
Although  the  ultimate  responsibility  for 
the  RFP  is  between  the  PM  and  the  primary 
contracting  officer  (PCO),  the  Deputy  for 
T&E  is  responsible  for  creating  several 
sections.  These  sections  include  the  test 
schedule,  test  program  funding  (projec¬ 
tions),  test  data  requirements  for  the  pro¬ 
gram  (test  reports,  plans,  procedures. 
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quick-look  reports,  etc.),  the  test  section  of 
the  Statement  of  Work  (SOW),  portions  of 
the  Acquisition  Plan,  Information  for  Pro¬ 
posal  Preparation  (IFPP),  and  (if  a  joint 
acquisition  program)  the  Joint  Operational 
Requirements  Document  (JORD). 

4.3.1  Memorandums 

Early  in  the  program,  another  task  of  the 
Deputy  for  T&E  is  the  arrangement  of  any 
Memorandums  of  Agreement  or  Under¬ 
standing  (MOA/MOU)  between  Services, 
NATO  countries,  test  organizations,  etc., 
which  outline  the  responsibilities  of  each 
organization.  The  RFP/SOW  outline  con¬ 
tractor/government  obligations  and  ar¬ 
rangements  on  the  access  and  use  of  test 
facilities  (contractor  or  government 
owned). 

4.3.2  Test  Data  Management 

The  Deputy  for  T&E  may  have  approval 
authority  for  all  contractor-created  test 
plans,  procedures  and  reports.  The  Deputy 
for  T&E  must  have  access  to  all  contractor 
testing  and  test  results,  and  the  Deputy  for 
T&E  is  responsible  for  disseminating  the 
results  to  government  agencies  that  need 
this  data.  Additionally,  the  Deputy  for  T&E 
creates  report  formats  and  time  lines  for 
contractor  submittal,  government  approval, 
etc. 

The  data  requirements  for  the  entire  test 
program  are  outlined  in  the  Contract  Data 
Requirements  List  (CDRL).  The  Eteputy  for 
T&E  should  review  the  Acquisition  Man¬ 
agement  Systems  and  Data  Requirements 
Control  List  (AMSDL),  Department  of  De¬ 
fense  (DoD)  5010.12-L,  for  relevant  test  data 
item  descriptions  (DIDs).  (Examples  can  be 
found  in  Appendix  C.)  The  Deputy  for  T&E 
provides  input  to  this  section  of  the  RFP 
early  in  the  program.  The  Deputy  for  T&E 
ensures  that  the  office  of  the  Deputy  for 


T&E  and  all  associated  test  organizations 
requiring  the  information  receive  the  test 
documentation  on  time.  Usually,  the  con¬ 
tractor  sends  the  data  packages  directly  to 
the  Deputy  for  T&E,  who,  in  turn,  has  a 
distribution  list  trimmed  to  the  minimum 
number  of  copies  for  agencies  needing  that 
information  to  perform  their  mission  and 
oversight  responsibilities.  It  is  important 
for  the  Deputy  for  T&E  to  use  an  integrated 
test  program  and  request  contractor  test 
plans  and  procedures  well  in  advance  of 
the  actual  test  performance  to  ensure  that 
the  office  of  the  Deputy  for  T&E  has  time  to 
approve  the  procedures  or  implement 
modifications. 

Conversely,  the  Deputy  for  T&E  must  re¬ 
ceive  the  test  results  and  reports  on  time  to 
enable  the  office  of  the  Deputy  for  T&E,  the 
PM  and  higher  authorities  to  make  pro¬ 
gram  decisions.  Further,  the  data  received 
should  be  tailored  to  provide  the  minimum 
information  needed.  The  Deputy  for  T&E 
must  be  aware  that  data  requirements  in 
excess  of  the  minimum  needed  will  lead  to 
an  unacceptable  increase  in  overall  pro¬ 
gram  cost.  For  data  that  is  needed  quickly 
and  informally  (at  least  initially),  the  Deputy 
for  T&E  can  request  Quick-Look  Reports 
that  give  test  results  immediately  after  test 
performance.  The  Deputy  for  T&E  is  also 
responsible  for  coordinating  with  the  con¬ 
tractor  on  all  report  formats  (the  in-house 
contractor  format  is  acceptable  in  most 
cases). 

The  contract  must  specify  the  data  the  con¬ 
tractor  will  supply  the  operational  test 
agency  (OTA).  Unlike  development  test 
and  evaluation  (DT&E),  the  contractor  will 
not  be  making  the  operational  test  and 
evaluational  (OT&E)  plans,  procedures  or 
reports.  These  documents  are  the  responsi¬ 
bility  of  the  OTA.  The  PMO  Deputy  for 
T&E  should  include  the  OTA  on  the  distri¬ 
bution  list  for  all  test  documents  that  are  of 
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concern  during  the  DT&E  phase  of  testing 
so  they  will  be  informed  of  test  item  progress 
and  previous  testing.  In  this  way,  the  OTA 
willbeinformed  when  developing  their  own 
test  plans  and  procedures  for  OT&E.  In 
fact,  OTA  representatives  should  attend 
the  CDRL  Review  Board  and  provide  the 
PMO  with  a  list  of  the  types  of  documents 
the  OTA  will  need.  The  Deputy  for  T&E 
should  coordinate  the  test  sections  of  this 
data  list  with  the  OTA  and  indicate  con¬ 
cerns  at  that  meeting.  All  contractor  test 
reports  should  be  made  available  to  the 
OTA.  In  return,  the  Deputy  for  T&E  must 
stay  informed  of  all  OTA  activities,  under¬ 
stand  their  test  procedures,  and  plan  and 
receive  their  test  reports.  Unlike  DT&E,  the 
PMO  Deputy  for  T&E  will  not  have  report 
or  document  approval  authority  for  OT&E 
items  as  he/she  does  over  contractor  docu¬ 
mentation.  The  Deputy  for  T&E  is  always 
responsible  for  keeping  the  PM  informed 
of  OT&E  results. 

4.3.3  Test  Schedule  Formulation 

A  very  important  task  the  Deputy  for  T&E 
has  during  the  creation  of  the  RFP  is  the  test 
program  schedule.  Initially,  the  PM  will 
need  contractor  predictions  of  the  hard¬ 
ware  (and  software  in  some  cases)  avail¬ 
ability  dates  for  models,  prototypes, 
mockups,  full-scale  models,  etc.,  once  the 
contract  is  awarded.  The  Deputy  for  T&E 
uses  this  information  to  create  a  realistic 
front-end  schedule  of  the  in-house  testing 
the  contractor  will  conduct  before  govern¬ 
ment  testing  (development  testing  (DT)  and 
operational  testing  (OT)).  Then,  a 
"strawman"  schedule  is  developed  upon 
which  the  government  DT  and  OT  sched¬ 
ules  can  be  formulated  and  contractor  sup¬ 
port  requirements  determined .  The  Deputy 
for  T&E  can  use  past  experience  in  testing 
similar  weapon  systems/acquisition  items 
or  contract  test  organizations  that  have  the 
required  experience  to  complete  the  entire 


test  schedule.  Since  the  test  schedule  is  a 
critical  contractual  item,  contractor  input  is 
very  important.  The  test  schedule  will  nor¬ 
mally  become  an  item  for  negotiation  once 
the  RFP  is  released,  and  the  contractor’s 
proposal  is  received.  Attention  must  be 
given  to  ensuring  the  test  schedule  is  not 
too  success-oriented  that  test  failures  cause 
serious  program  delays  for  either  the  gov¬ 
ernment  test  agencies  or  the  contractor. 

Another  important  early  activity  the  Deputy 
for  T&E  must  accomplish  is  to  coordinate 
the  OT&E  test  schedule.  Since  the  contrac¬ 
tor  may  be  required  to  provide  support,  the 
OT&E  test  support  may  need  to  be  contrac¬ 
tually  agreed  upon  before  contract  award. 
Sometimes,  the  Deputy  for  T&E  can  formu¬ 
late  a  strawman  schedule  (based  on  previ¬ 
ous  experience)  and  present  this  schedule 
to  the  operational  test  representative  at  the 
initial  T&E  Integrated  Product  Team  (IPT) 
meeting  for  review;  or  the  Deputy  for  T&E 
can  contact  the  OTA  and  arrange  a  meeting 
to  discuss  the  new  program.  In  the  meet¬ 
ing,  time  requirements  envisioned  by  OTA 
can  be  discussed.  Input  from  that  meeting 
then  goes  into  the  RFP  and  to  the  PM.  The 
test  schedule  must  allow  time  for  DT&E 
testing  and  OT&E  testing  when  testing  is 
not  combined  or  test  assets  are  not  limited. 
Before  set-up  of  initial  operational  test  and 
evaluation  (lOT&E),  certification  of  readi¬ 
ness  for  lOT&E  may  require  a  time  gap  for 
review  of  DT&E  test  results  and  refurbish¬ 
ment  or  corrections  of  deficiencies  discov¬ 
ered  during  DT&E,  etc.  The  test  schedule 
for  DT&E  should  not  be  so  "success-ori¬ 
ented"  that  the  lOT&E  test  schedule  is  ad¬ 
versely  impacted,  not  allowing  enough  time 
for  adequate  operational  testing  or  the  re¬ 
porting  of  lOT&E  results.  For  example,  if 
the  DT&E  schedule  slips  six  months,  the 
OT&E  schedule  and  milestone  decision 
should  slip  also.  The  lOT&E  should  not  be 
shortened  just  to  make  a  milestone  deci¬ 
sion  date. 
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4.3.4  Programmatic  Environmental 
Analysis 

The  PMO  personnel  should  be  sensitive  to 
the  potential  environmental  consequences 
of  system  materials,  operations  and  dis¬ 
posal  requirements.  Public  Laws  (Title  40, 
Code  of  Federal  Regulations,  Parts  1500- 
1508;  National  Environmental  Policy  Act 
(NEPA)  Regulations;  Executive  Order 
12114,  Environmental  Effects  Abroad  of 
Major  Federal  Actions;  DoD  5000  series; 
etc.)  require  analysis  of  hazardous  materi¬ 
als  and  appropriate  mitigation  measures 
during  each  acquisition  phase.  As  stated  in 
DoD  5000.2-R,  "environmental  regulations 
are  a  source  of  external  constraints  that 
must  be  identified  and  integrated  into  pro¬ 
gram  execution." 

Litigations  resulting  in  personal  fines  and 
imprisonment  successfully  executed 
against  government  employees  have  raised 
the  environmental  awareness  at  test  ranges 
and  facilities.  Environmental  Impact  State¬ 
ments  (supported  by  long,  thorough  stud¬ 
ies  and  public  testimony)  or  Environmen¬ 
tal  Analysis  and  Assessments  are  generally 
required  before  any  system  testing  can  be 
initiated. 

4.4  PMO/CONTRACTOR 
TEST  MANAGEMENT 

The  PMO  will,  in  most  cases,  have  a 
contractor  test  section  counterpart.  With 
this  counterpart,  the  Deputy  for  T&E 
works  out  the  detailed  test  planning,  cre¬ 
ation  of  schedules,  etc.,  for  the  entire  test 
program.  The  PMO  uses  input  from  all 
sources  (contracts,  development  test 
agencies,  operational  test  agencies,  higher 
headquarters,  etc.)  to  formulate  the  test 
program's  length,  scope  and  necessary 
details.  The  Deputy  for  T&E  ensures  that 
the  RFP  reflects  the  test  program  envi¬ 
sioned  and  the  contractor's  role  in  the 


acquisition  process.  The  Deputy  for  T&E 
also  ensures  the  RFP  includes  provisions 
for  government  attendance  at  contractor's 
tests  and  that  all  contractor  test  results  are 
provided  to  the  government. 

After  the  RFP  has  been  issued  and  the 
contractor  has  responded,  the  proposal  is 
reviewed  by  the  PMO.  The  Deputy  for  T&E 
is  responsible  for  performing  a  technical 
evaluation  on  the  test  portions  of  the  pro¬ 
posal.  In  this  technical  evaluation,  the 
Deputy  for  T&E  compares  the  proposal  to 
the  SOW,  test  schedule,  IFPP,  etc.,  and  re¬ 
views  the  contractor's  cost  of  each  testing 
item.  This  is  an  iterative  process  of  refining, 
clarifying  and  modifying  that  will  ensure 
the  final  contract  between  the  PMO  and  the 
prime  contractor  (subcontractors)  contains 
all  test-related  tasks  and  is  priced  within 
scope  of  the  proposed  test  program.  Once 
technical  agreement  on  the  contractor's  tech¬ 
nical  approach  is  reached,  the  Deputy  for 
T&E  is  responsible  for  giving  inputs  to  the 
government  contracting  officer  during  con¬ 
tract  negotiations.  The  contracting  officer- 
requested  contract  deliverables  are  assigned 
contract  line  item  numbers  (CLINs),  which 
are  created  by  the  Deputy  for  T&E.  This 
will  ensure  the  contractor  delivers  the  re¬ 
quired  performances  at  specified  intervals 
during  the  life  of  the  contract.  Usually, 
there  will  be  separate  contracts  for  devel¬ 
opment  and  production  of  the  acquisition 
item.  For  each  type  of  contract,  the  Deputy 
for  T&E  has  the  responsibility  to  provide 
the  PCO  and  PM  with  the  test  and  evalua¬ 
tion  input. 

4.5  INTEGRATED  PRODUCT 
TEAMS  FOR  TEST 
AND  EVALUATION 

Before  the  final  version  of  the  RFP  is  cre¬ 
ated,  the  Deputy  for  T&E  will  form  an  IPT, 
a  test  planning/integration  working  group. 
This  group  includes  the  operational  test 
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agency,  development  test  agency,  organi¬ 
zations  that  may  be  jointly  acquiring  the 
same  system,  the  test  supporting  agencies, 
operational  users,  and  any  other  organiza¬ 
tions  that  will  be  involved  in  the  test  pro¬ 
gram  by  providing  test  support  or  by  con¬ 
ducting,  evaluating  or  reporting  on  testing. 
The  functions  of  the  groups  are  to:  facilitate 
the  use  of  testing  expertise,  instrumenta¬ 
tion,  facilities,  simulations  and  models;  in¬ 
tegrate  test  requirements;  accelerate  the 
TEMP  coordination  process;  resolve  test 
cost  and  scheduling  problems;  and  pro¬ 
vide  a  forum  to  ensure  T&E  of  the  system  is 
coordinated.  The  existence  of  a  test  coordi¬ 
nating  group  does  not  alter  the  responsi¬ 
bilities  of  any  command  or  headquarters; 
and  in  the  event  of  disagreement  within  a 
group,  the  issue  is  resolved  through  the 
normal  command/staff  channels.  In  later 
meetings,  the  contractor  participates  in  this 
test  planning  group;  however,  the  contrac¬ 
tor  may  not  be  selected  by  the  time  the  first 
meetings  are  held. 

The  purposes  of  these  meetings  are  to  re¬ 
view  and  assist  in  the  development  of  early 
test  documentation,  the  TEMP,  and  to  agree 
on  basic  test  program  schedules,  scope, 
support,  etc.  The  TEMP  serves  as  the  top- 
level  test  management  document  for  the 
acquisition  program,  being  updated  as  the 
changing  program  dictates. 

4.6  TEST  PROGRAM 
FUNDING/BUDGETING 

The  PMO  must  identify  funds  for  testing 
very  early  so  that  test  resources  can  be 
obtained.  The  Deputy  for  T&E  uses  the 
acquisition  schedule,  TEMP  and  other  pro¬ 
gram  and  test  documentation  to  identify 
test  resource  requirements.  The  Deputy  for 
T&E  coordinates  these  requirements  with 
the  contractor  and  government  organiza¬ 
tions  that  have  the  test  facilities  to  ensure 
their  availability  for  testing.  The  Deputy 


for  T&E  ensures  that  test  costs  include  con¬ 
tractor  and  government  test  costs.  The 
contractor's  test  costs  are  normally  out¬ 
lined  adequately  in  his  proposal;  however, 
the  government  test  ranges,  instrumenta¬ 
tion  and  test-support  resource  costs  must 
be  determined  by  other  means.  Usually, 
the  Deputy  for  T&E  contacts  the  test  orga¬ 
nization  and  outlines  the  test  program  re¬ 
quirements;  and  the  test  organization  sends 
the  program  office  an  estimate  of  the  test 
program  costs.  The  Deputy  for  T&E  then 
obtains  cost  estimates  from  all  test  sources 
that  the  Deputy  for  T&E  anticipates  using 
and  supplies  this  information  to  the  PM. 
The  Deputy  for  T&E  must  also  ensure  that 
any  program  funding  reductions  are  not 
absorbed  entirely  by  the  test  program.  Some 
cutbacks  may  be  necessary  and  allowable; 
but  the  test  program  must  supply  the  PM, 
other  defense  decision-making  authori¬ 
ties,  and  the  Congress  with  enough  infor¬ 
mation  to  make  program  milestone  deci¬ 
sions. 

The  Deputy  for  T&E  provides  the  PM  esti¬ 
mates  of  PMO  test  program  costs  to  con¬ 
duct  lOT&E.  This  funding  includes  con¬ 
tractor  and  government  test  support  for 
which  the  program  office  directly  or  indi¬ 
rectly  will  be  responsible.  Since  Service 
OTAs  fund  differently,  program  office 
funding  for  conducting  OT&E  varies.  The 
Deputy  for  T&E  must  determine  these  costs 
and  inform  the  PM. 


4.7  TECHNICAL 
REVIEWS,  DESIGN 
REVIEWS  AND  AUDITS 

The  role  of  the  Deputy  for  T&E  changes 
slightly  during  the  contractor's  technical 
reviews,  design  reviews,  physical  and 
functional  configuration  audits,  etc.  Usu¬ 
ally,  the  Deputy  for  T&E  plans,  directs  or 
monitors  government  testing;  however,  in 
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the  reviews  and  audits,  the  Deputy  exam¬ 
ines  the  contractor's  approach  to  the  test 
problem  and  evaluates  the  validity  of  the 
process  and  the  accuracy  of  the  contractor’s 
results.  The  Deputy  for  T&E  uses  personal 
experience  and  background  in  test  and 
evaluation  to  assess  whether  the  contractor 
did  enough  or  too  little  testing;  whether  the 
tests  were  biased  in  any  way;  and  if  they 
followed  a  logical  progression  using  the 
minimum  of  time,  effort  and  funds.  If  the 
Deputy  for  T&E  finds  any  discrepancies, 
the  Deputy  must  inform  the  contractor,  the 
PM  and  the  PCO  to  validate  the  conclu¬ 
sions  before  effecting  corrections.  Each  type 
of  review  or  audit  will  have  a  different 
focus/orientation,  but  the  Deputy  for  T&E 
will  always  be  concerned  with  the  testing 
process  and  how  it  is  carried  out.  After  each 
review,  the  Deputy  for  T&E  should  always 
document  all  observations  for  future  refer¬ 
ence. 

4.8  CONTRACTOR 
TESTING 

The  Deputy  for  T&E  is  responsible  for  en¬ 
suring  that  contractor-conducted  tests  are 
monitored  by  the  government.  The  Deputy 
for  T&E  must  also  be  given  access  to  all 
contractor  internal  data,  test  results  and 
test  reports  related  to  the  acquisition  pro¬ 
gram.  Usually,  the  contract  requires  that 
government  representatives  be  informed 
ahead  of  time  of  any  (significant  or  other¬ 
wise)  testing  the  contractor  conducts  so  the 
government  can  arrange  to  witness  certain 
testing  or  receive  results  of  the  tests.  Fur¬ 
ther,  the  contractor's  internal  data  should 
be  available  as  a  contract  provision.  The 
Deputy  for  T&E  must  ensure  that  gov¬ 
ernment  test  personnel  (DT&E/OT&E) 
have  access  to  contractor  test  results.  It 
would  be  desirable  to  have  all  testers 
observe  some  contractor  tests  to  help  de¬ 
velop  confidence  in  the  results  and  iden¬ 
tify  areas  of  risk. 


4.9  SPECIFICATIONS 

Within  the  program  office,  the  engineering 
section  is  usually  tasked  to  create  the  sys¬ 
tem  performance  specifications  for  release 
of  the  RFP.  The  contractor  is  then  tasked 
with  creating  the  specification  documenta¬ 
tion  called  out  by  the  contract,  which  will 
be  delivered  once  the  item /system  design 
is  formalized  for  production.  The  Deputy 
for  T&E  performs  an  important  function  in 
specification  formulation  by  reviewing  the 
specifications  to  determine  if  performance 
parameters  are  testable;  if  current,  state-of- 
the-art  technology  can  determine  (during 
the  DT&E  test  phase)  if  the  performance 
specifications  are  being  met  by  the  acquisi¬ 
tion  item;  or  if  the  specified  parameters  are 
too  "tight."  A  specification  is  too  "tight"  if 
the  requirements  (Sec  3)  are  impossible  to 
meet  or  demonstrate,  if  the  specification 
has  no  impact  on  the  form,  fit  or  function  of 
the  end-item,  the  system  it  will  become  a 
part  of  or  the  system  with  which  it  will 
interact.  The  Deputy  for  T&E  must  deter¬ 
mine  if  test  objectives  can  be  adequately 
formulated  from  those  specifications  that 
will  provide  thresholds  of  performance, 
minimum  and  maximum  standards  and 
reasonableoperating  conditions  for  theend- 
item's  final  mission  and  operating  environ¬ 
ment.  The  specifications  shape  the  devel¬ 
opment  test  and  evaluation  (DT&E)  testing 
scenario,  test  ranges,  test  support,  targets, 
etc.,  and  are  very  important  to  the  Deputy 
for  T&E. 

4.10  INDEPENDENT  TEST  AND 
EVALUATION  AGENCIES 

The  PMO  Deputy  for  T&E  does  not  have 
direct  control  over  government-owned  test 
resources,  test  facilities,  test  ranges,  test 
personnel,  etc.  Therefore,  the  Deputy  for 
T&E  must  depend  on  those  DT  or  OT  test 
organizations  controlling  them  and  stay 
involved  with  the  test  agency  activities. 
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•  Understand  the  policies 

•  Organize  for  T&E 

•  Keep  system  requirements  documents  current 

•  Agonize  over  system  thresholds 

•  Work  closely  with  the  operational  test  director 

•  Don’t  forget  about  operational  suitability 

•  Make  final  DT&E  a  rehearsal  for  lOT&E 

•  Prepare  interfacing  systems  for  your  lOT&E 

•  Manage  software  testing  closely 

•  Track  availability  of  test  resources  and  test 
support  personnel/facilities 

Source:  NAVSEATE 


Figure  4-1 .  Lessons  Learned  from  OT&E  for  the  PM 


The  amount  of  involvement  depends  on 
the  item  being  tested;  its  complexity,  cost 
and  characteristics;  the  length  of  time  for 
testing;  amount  of  test  funds;  etc.  Usually, 
the  "nuts  and  bolts"  detailed  test  plans  and 
procedures  are  written  by  the  test  organi¬ 
zations  controlling  the  test  resources  with 
input  and  guidance  from  the  Program  Of¬ 
fice  Deputy  for  T &E.  The  Deputy  for  T &E  is 
responsible  for  ensuring  that  the  tests  are 
performed  using  test  objectives  based  on 
the  specifications  and  that  the  requirements 
of  timeliness,  accuracy  and  minimal  costs 
are  met  by  the  test  program  design.  During 
the  testing,  the  Deputy  for  T&E  monitors 
test  results.  The  test  agencies  submit  a  copy 
of  their  report  to  the  Program  Office  at  the 
end  of  testing,  usually  to  the  Office  of  the 
Deputy  for  T&E.  The  Army  is  the  only 
Service  to  have  a  designated  independent 
evaluation  agency  which  provides  feed¬ 
back  to  the  program  office. 

4.11  PMO  RESPONSIBILITIES  FOR 
OPERATIONAL  TEST  AND 
EVALUATION  (OT&E) 

In  the  government  PMO,  there  should  be  a 
section  responsible  for  T&E.  Besides  be¬ 
ing  responsible  for  DT&E  support  to  the 
PM,  this  section  should  be  responsible 


for  program  coordination  with  the  OT&E 
agency  (Figure  4-1).  The  offices  of  the  sys¬ 
tems  engineer  or  the  Deputy  for  T&E  may 
be  designated  to  provide  this  support  to  the 
program  manager.  In  some  Services,  re¬ 
sponsibilities  of  the  Deputy  for  T&E  in¬ 
clude  coordination  of  test  resources  for  all 
phases  of  OT&E. 

4.11.1  Contract  Responsibilities 

The  Deputy  for  T&E  or  a  T&E  representa¬ 
tive  ensures  that  certain  sections  of  the  REP 
contain  sufficient  allowance  for  T&E  sup¬ 
port  by  contractors.  This  applies  whether 
the  contract  is  for  a  development  item,  a 
production  item  (limited  production,  such 
as  low  rate  initial  production  (LRIP)  or  full- 
rate  production)  or  the  enhancement/up¬ 
grade  of  portions  of  a  weapons  system. 
Where  allowed  within  the  law,  contractor 
support  for  OT&E  should  be  considered  to 
help  resolve  basic  issues  such  as  data  col¬ 
lection  requirements,  test  resources,  con¬ 
tractor  test  support  and  funding. 

In  the  overall  portion  of  the  RFP,  govern¬ 
ment  personnel,  especially  those  in  the 
operational  test  agencies,  must  be  guaran¬ 
teed  access  to  the  contractor's  development 
facilities,  particularly  during  the  DT&E 
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Phase.  Government  representatives  must 
be  allowed  to  observe  all  contractor  in- 
house  testing  and  have  access  to  test  data 
and  reports. 

4.11.2  Data  Requirements 

The  contract  must  specify  the  data  the  con¬ 
tractor  will  supply  the  OTA).  Unlike  DT&E, 
the  contractor  will  not  be  making  the  OT &E 
plans,  procedures  or  reports.  These  docu¬ 
ments  are  the  responsibility  of  the  OTA. 
The  PMO  Deputy  for  T&E  should  include 
the  OTA  on  the  distribution  list  for  all  test 
documents  that  are  of  concern  during  the 
DT&E  phase  of  testing  so  they  will  be  in¬ 
formed  of  test  item  progress  and  previous 
testing.  In  this  way,  the  OTA  will  be  in¬ 
formed  when  developing  their  own  test 
plans  and  procedures  for  OT&E.  In  fact, 
OTA  representatives  should  attend  the 
CDRL  Review  Board  and  provide  the  PMO 
with  a  list  of  the  types  of  documents  the 
OTA  will  need.  The  Deputy  for  T&E  should 
coordinate  the  test  sections  of  this  data  list 
with  the  OTA  and  indicate  concerns  at  that 
meeting.  All  contractor  test  reports  should 
be  made  available  to  the  OTA.  In  return,  the 
Deputy  for  T&E  must  stay  informed  of  all 
OTA  activities,  understands  their  test  pro¬ 
cedures  and  plans  and  receives  their  test 
reports.  Unlike  DT&E,  the  PMO  Deputy  for 
T&E  will  not  have  report  or  document  ap¬ 
proval  authority  as  the  Deputy  for  T&E 
does  over  contractor  documentation.  The 
Deputy  for  T&E  is  always  responsible  for 
keeping  the  PM  informed  of  OT&E  results. 

4.11.3  Test  Schedule 

Another  important  early  activity  the  Deputy 
for  T&E  must  accomplish  is  to  coordinate 
the  OT&E  test  schedule.  Since  the  contrac¬ 
tor  may  be  required  to  provide  support,  the 
OT&E  test  support  may  need  to  be  contrac¬ 
tually  agreed  upon  before  contract  award. 
Sometimes,  the  Deputy  for  T&E  can  for¬ 


mulate  a  strawman  schedule  (based  on 
previous  experience)  and  present  this 
schedule  to  the  operational  test  representa¬ 
tive  at  the  initial  test  planning  working 
group  for  review;  or  the  Deputy  for  T&E 
can  contact  the  OTA  and  arrange  a  meeting 
to  discuss  the  new  program.  In  the  meet¬ 
ing,  time  requirements  envisioned  by  OTA 
can  be  discussed.  Input  from  that  meeting 
then  goes  into  the  REP  and  to  the  PM.  The 
test  schedule  must  allow  time  for  DT&E 
testing  and  OT&E  testing  if  testing  is  not 
combined  or  test  assets  are  limited.  Before 
set-up  of  initial  lOT&E,  certification  of  readi¬ 
ness  for  lOT&E  may  require  a  time  gap  for 
review  of  DT&E  test  results  and  refurbish¬ 
ment  or  corrections  of  deficiencies  discov¬ 
ered  during  DT&E,  etc.  The  test  schedule 
for  DT&E  should  not  be  so  "success-ori¬ 
ented"  that  the  lOT&E  test  schedule  is  ad¬ 
versely  impacted,  not  allowing  enough  time 
for  adequate  operational  testing  or  the  re¬ 
porting  of  lOT&E  results.  For  example,  if 
the  DT&E  schedule  slips  six  months,  the 
OT&E  schedule  and  milestone  decision 
should  slip  also.  The  lOT&E  should  not  be 
shortened  just  to  make  a  milestone  deci¬ 
sion  date. 

4.11.4  Contractor  Support 

The  Deputy  for  T&E  provides  all  T&E  in¬ 
put  to  the  RFP/SOW.  The  Deputy  for  T&E 
must  determine,  before  the  beginning  of 
the  program  acquisition  phase,  whether 
the  contractor  will  be  involved  in  support¬ 
ing  OT&E  and,  if  so,  to  what  extent.  Ac¬ 
cording  to  Title  10,  U.S.C.,  the  system  con¬ 
tractor  can  only  be  involved  in  the  conduct 
of  lOT&E  if,  once  the  item  is  fielded,  tactics 
and  doctrine  say  the  contractor  will  be 
providing  support  or  operating  that  item 
during  combat.  If  not,  no  system  contractor 
support  is  allowed  during  OT&E.  Before 
lOT&E;  however,  the  contractor  may  be 
tasked  with  providing  training,  training 
aids  and  handbooks  to  Service  training 
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cadre  so  they  can  train  the  lOT &E  users  and 
maintenance  personnel.  In  addition,  the 
contractor  must  be  required  to  provide  suf¬ 
ficient  spare  parts  for  the  operational  main¬ 
tenance  personnel  to  maintain  the  test  item 
while  undergoing  operational  testing.  These 
support  items  must  be  agreed  upon  by  the 
PMO  and  OTA  and  must  contractually  bind 
the  contractor.  If,  however,  the  contractor 
will  be  required  to  provide  higher-level 
maintenance  of  the  item  for  the  duration  of 
the  lOT&E,  data  collection  on  those  func¬ 
tions  will  be  delayed  until  a  subsequent 
follow-on  operational  test  and  evaluation 
(FOT&E). 

4.11.5  Operational  OT&E  Funding 

The  Deputy  for  T&E  provides  the  PM  esti¬ 
mates  of  PMO  test  program  costs  to  con¬ 
duct  lOT&E.  This  binding  includes  con¬ 
tractor  and  government  test  support  for 
which  the  program  office  directly  or  indi¬ 
rectly  will  be  responsible.  Since  Service 
OTAs  fund  differently,  program  office 
funding  for  conducting  OT&E  varies.  The 
Deputy  for  T&E  must  determine  these  costs 
and  inform  the  PM. 

4.11.6  Statement  of  Work 

One  of  the  most  important  documents  re¬ 
ceiving  input  from  the  Deputy  for  T&E  is 
the  SOW.  The  Deputy  for  T&E  must  outline 
all  required  or  anticipated  contractor  sup¬ 
port  for  DT&E  and  OT&E.  This  document 
outlines  data  requirements,  contractor-con¬ 
ducted  or  supported  testing,  government 
involvement  (access  to  contractor  data,  tests 
and  results),  operational  test  support,  and 
any  other  specific  test  requirements  the 
contractor  will  be  tasked  to  perform  during 
the  duration  of  the  contract. 

4.11.7  Test  and  Evaluation  Master 
Plan  (TEMP) 

The  TEMP  should  be  updated  regularly  by 
the  OTA.  The  Deputy  for  T&E  is  responsible 


for  managing  the  TEMP  throughout  the 
test  program.  The  OTA  usually  is  tasked  to 
complete  the  operational  test  section  of  the 
TEMP  and  outline  their  proposed  test  pro¬ 
gram  through  all  phases  of  OT&E.  It  is 
important  to  keep  the  TEMP  updated  regu¬ 
larly  so  that  test  organizations  involved  in 
OT&E  understand  the  scope  of  their  test 
support.  Further,  if  any  upgrades,  improve¬ 
ments  or  enhancements  to  the  fielded 
weapon  system  occur,  the  TEMP  must  be 
updated  or  a  new  one  created  to  outline 
new  DT  and  OT  requirements. 

4.11.8  Program  Management  Office 
Support  for  OT&E 

Even  though  operational  testing  is  per¬ 
formed  by  an  independent  organization, 
the  PM  plays  an  important  role  in  its  plan¬ 
ning,  reporting  and  funding.  The  PM  must 
coordinate  program  activities  with  the  test 
community,  especially  the  operational  test 
agencies.  The  PM  ensures  that  testing  can 
address  the  critical  issues,  and  provides 
feedback  from  OT&E  testing  activities  to 
contractors. 

At  each  milestone  review,  the  PM  is  re¬ 
quired  to  brief  the  decision  authority  on  the 
testing  planned  and  completed  on  the  pro¬ 
gram.  It  is,  therefore,  important  that  PMO 
personnel  have  a  good  understanding  of 
the  test  program  and  that  they  work  with 
the  operational  test  community.  This  will 
ensure  OT&E  is  well-planned  and  adequate 
resources  are  available.  The  PMO  should 
involve  the  test  community  by  organizing 
test  coordinating  groups  at  program  initia¬ 
tion  and  by  establishing  channels  of  com¬ 
munication  between  the  PMO  and  the  key 
test  organizations.  The  PMO  can  often 
avoid  misunderstandings  by  aggressively 
monitoring  the  system  testing  and  pro¬ 
viding  up-to-date  information  to  key  per¬ 
sonnel  in  the  Office  of  the  Secretary  of 
Defense  and  the  Services.  The  PMO  staff 
should  keep  appropriate  members  of  the 
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test  community  well-informed  concerning 
system  problems  and  the  actions  taken  by 
the  PMO  to  correct  them.  The  PMO  must 
assure  that  contractor  and  government 
DT&E  supports  the  decision  to  certify  the 
system's  readiness  for  lOT&E. 

4.11.9  Support  for  Initial  Operational 
Test  and  Evaluation 

For  lOT&E,  the  Deputy  for  T&E  must  en¬ 
sure  the  contract  portions  adequately  cover 
the  scope  of  testing  as  outlined  by  the  op¬ 
erational  test  agency.  The  program  office 
may  want  to  provide  an  observer  to  repre¬ 
sent  the  Deputy  for  T&E  during  the  actual 
testing.  The  Deputy  for  T&E  involvement 
in  lOT&E  will  be  to  monitor  and  coordi¬ 
nate;  the  Deputy  for  T&E  will  keep  the  PM 
informed  of  progress  and  problems  that 
arise  during  testing  and  will  monitor  re¬ 
quired  PMO  support  to  the  test  organiza¬ 
tion.  Also,  enough  LRIP  items  must  be 
manufactured  to  run  a  complete  and  ad¬ 
equate  OT&E  program.  For  problems  re¬ 
quiring  program  office  action,  the  Deputy 
for  T&E  will  be  the  point  of  contact. 

The  Deputy  for  T&E  will  be  concerned  with 
lOT&E  of  the  LRIP  units  after  a  limited 
number  are  produced.  The  lOT&E  must  be 
closely  monitored  so  that  a  full-rate  pro¬ 
duction  decision  can  be  made.  As  in  the 
operational  assessments,  the  Deputy  for 
T&E  will  be  monitoring  test  procedures 
and  results  and  keeping  the  PM  informed. 
If  the  item  does  not  succeed  during  lOT&E, 
a  new  process  of  DT&E  or  a  modification 
may  result;  and  the  Deputy  for  T&E  will  be 
involved  (as  in  any  new  programs  incep¬ 
tion).  If  the  item  passes  lOT&E  testing  and 
is  produced  at  full  rate,  the  Deputy  for  T&E 
will  be  responsible  for  ensuring  that  testing 
of  those  production  items  is  adequate  to 
ensure  that  the  end  items  physically  and 
functionally  resemble  the  development 
items. 


4.11.10  FOT&E  and  Modifications, 
Upgrades,  Enhancements, 
or  Additions 

During  FOT&E,  the  Deputy  for  T&E  moni¬ 
tors  the  testing;  the  contractor  is  usually 
not  involved.  The  Deputy  for  T&E  should 
receive  any  reports  generated  by  the  opera¬ 
tional  testers  during  this  time.  Any  defi¬ 
ciencies  noted  during  FOT&E  should  be 
evaluated  by  the  PMO,  which  may  de¬ 
cide  to  incorporate  upgrades,  enhance¬ 
ments  or  additions  to  the  current  system. 
If  the  PM  and  the  engineering  section  of 
the  program  office  design  or  develop 
modifications  that  are  incorporated  into 
the  weapon  system  design,  additional 
FOT&E  may  be  required. 

Once  a  weapon  system  is  fielded,  portions 
of  that  system  may  become  obsolete,  inef¬ 
fective  or  deficient  and  may  need  replac¬ 
ing,  upgrading  or  enhancing  to  ensure  the 
weapon  system  meets  current  and  future 
requirements.  The  Deputy  for  T&E  plays  a 
vital  role  in  this  process.  Modifications  to 
existing  weapon  systems  may  be  managed 
as  an  entire  newly  acquired  weapon  sys¬ 
tem.  However,  since  these  are  changes  to 
existing  systems,  the  Deputy  for  T&E  is 
responsible  for  determining  if  these  en¬ 
hancements  degrade  the  existing  system, 
are  compatible  with  its  interfaces  and  func¬ 
tions  and  whether  nondevelopment  items 
(NDIs)  require  retest  or  the  entire  weapon 
system  needs  reverification.  The  Deputy 
for  T&E  must  plan  the  test  program's  fund¬ 
ing,  schedule,  test  program  and  contract 
provisions  with  these  items  in  mind.  A  new 
TEMP  may  have  to  be  generated  or  the 
original  weapon  system  TEMP  modified 
and  recoordinated  with  the  test  organiza¬ 
tions.  The  design  of  the  DT&E  and  FOT&E 
program  usually  requires  coordination  with 
the  engineering,  contracting  and  program 
management  sections  of  the  program  of¬ 
fice. 


4-10 


4.11.11  Test  Resources 

During  all  phases  of  OT,  the  Deputy  for 
T&E  must  coordinate  with  the  operational 
testers  to  ensure  they  have  the  test  articles 
needed  to  accomplish  their  mission.  Test 
resources  will  be  either  contractor  provided 
or  government  provided.  The  contractor 
resources  must  be  covered  in  the  contract, 
whether  in  the  development  contract  or  the 
production  contract.  Government  test  re¬ 
sources  needed  are  determined  by  the  op¬ 
erational  testers.  They  usually  coordinate 
the  test  ranges,  test  support  and  the  user 
personnel  for  testing.  The  PM  programs 
funding  for  his  support  of  OT.  Funding  for 
Navy  operational  evaluation  (OPEVAL)  is 
identified  in  the  TEMP  and  funded  in  the 
PMO's  budget.  Other  Services  allow  the 
OTAs  to  develop  and  manage  their  own 
budget  for  operational  testing.  The  OTAs 
then  obligate  funds  for  test  ranges,  instru¬ 
mentation,  etc.,  according  to  their  opera¬ 
tional  test  plans. 

4.12  SUMMARY 

Staffing  requirements  in  the  PMO  vary 
with  the  program  phase  and  the  T&E 
workload.  Test  and  evaluation  expertise 


is  essential  in  the  early  planning  stages 
but  can  be  provided  through  matrix  sup¬ 
port.  The  Deputy  for  T&E  may  be  subor¬ 
dinate  to  the  chief  engineer  in  early  phases 
but  should  become  a  separate  staff  ele¬ 
ment  after  Milestone  (MS)  11.  Changing 
of  critical  players  can  destroy  established 
working  relationships  and  abrogate  prior 
agreements  if  continuity  is  not  maintained. 
The  PMO  management  of  T&E  must  pro¬ 
vide  for  an  integrated  focus  and  a  smooth 
transition  from  one  staff-support  mode  to 
the  next. 

The  PMO  should  be  proactive  in  its  rela¬ 
tions  with  the  Service  operational  testing 
agency.  There  are  many  opportunities  to 
educate  the  OTA  on  system  characteris¬ 
tics  and  expected  performance.  Early  OTA 
input  to  design  considerations  and  require¬ 
ments  clarification  can  reduce  downstream 
surprises.  Operational  testing  is  an  essen¬ 
tial  component  of  the  system  development 
and  decision-making  process.  It  can  be  used 
to  facilitate  system  development  or  may 
become  an  impediment.  In  many  cases,  the 
PMO  attitude  toward  operational  testing 
and  the  OTA  will  influence  which  role  the 
OTA  assumes. 
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TEST-RELATED  DOCUMENTATION 


5.1  INTRODUCTION 

Ehiring  the  course  of  a  defense  acquisition 
program,  many  documents  are  developed 
that  have  significance  for  those  responsible 
for  testing  and  evaluating  the  system.  This 
chapter  is  designed  to  provide  background 
on  some  of  these  documents. 

As  Figure  5-1  shows,  test-related  documen¬ 
tation  spans  a  broad  range  of  materials.  It 
includes  requirements  documentation  such 
as  the  Mission  Need  Statement  (MNS);  pro¬ 
gram  decision  documentation  such  as  Ac¬ 
quisition  Decision  Memorandum  (ADM) 
with  exit  criteria;  and  program  manage¬ 
ment  documentation  such  as  the  Acquisi¬ 
tion  Strategy,  Baseline  documentation,  the 
Technical  Management  Plan,  the  logistics 
support  planning  and  the  Test  and  Evalua¬ 
tion  Master  Plan  (TEMP).  Of  importance  to 
the  program  managers  (PM)  and  to  test  and 
evaluation  (T&E)  managers  are  additional 
test  program  documents  such  as  specific 
test  designs,  test  plans,  outline  test  plans/ 
test  program  outlines,  evaluation  plans  and 
test  reports.  This  chapter  concludes  with  a 
description  of  the  End-of-Test  Phase  and 
Beyond  Low  Rate  Initial  Production 
(BLRIP)  Reports,  and  two  special-purpose 
T&E  status  reports  that  are  used  to  support 
the  milestone  decision  process. 

5.2  REQUIREMENTS 
DOCUMENTATION 

5.2.1  Continuing  Mission  Area 
Analyses 

As  indicated  in  CJCSI  3170.01  (dated 
13  Jxme  1997),  the  Services  are  required  to 


conduct  continuing  mission  analyses  of 
their  assigned  areas  of  responsibility. 
These  Mission  Area  Analyses  (M  A  A)  may 
result  in  recommendations  to  initiate  new 
acquisition  programs  to  reduce  or  elimi¬ 
nate  operational  deficiencies.  If  a  need 
cannot  be  met  through  changes  in  tactics, 
strategy,  doctrine  or  training  and  a  mate¬ 
riel  solution  is  required,  the  needed  capa¬ 
bility  is  described  first  in  an  MNS  and 
then  in  the  Operational  Requirement 
Document  (ORD).  When  the  cost  of  a 
proposed  acquisition  program  is  esti¬ 
mated  to  exceed  $355  million  for  research, 
development,  test  and  evaluation  or 
$2,135  billion  for  procurement  (FY 1996$), 
it  is  considered  a  major  defense  acquisi¬ 
tion  program  and  requires  an  MNS.  The 
MNS  is  completed  at  the  beginning  of  a 
program  and  reviewed  to  evaluate  neces¬ 
sary  system  modifications  periodically. 

5.2.2  Mission  Need  Statement 
(MNS) 

The  MNS  is  a  short,  nonsystem-specific 
statement  of  operational  capability  need 
prepared  by  any  Department  of  Defense 
(DoD)  component  focusing  on  a  specific 
mission  area  need  or  deficiency.  Service 
validation  and,  for  those  potential  Acqui¬ 
sition  Category  (ACAT)  I  Programs,  re¬ 
view  and  validation  by  the  Joint  Require¬ 
ments  Oversight  Council  (JROC)  results 
in  forwarding  of  the  MNS  to  the  mile¬ 
stone  (MS)  decision  authority  for  MS  0 
consideration.  The  document’s  content 
and  format  (CJCSI  3170.01)  includes: 
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Figure  5-1.  Test-Related  Documentation 


•  Identification  of  the  applicable  Defense 
Planning  Guidance  Element; 

•  Mission  and  threat  analyses  —  need 
defined  in  terms  of  mission,  objectives 
and  general  capabilities; 

•  Nonmateriel  alternatives  —  tactics, 
doctrine,  organization  and  training; 

•  Potential  materiel  alternatives  — 
nondevelopment  item  (NDI),  allied,  in- 
ter-Service,  and  new; 

•  Constraints  by  infrastructure,  trea¬ 
ties  and  environments. 

The  MNS  and  other  requirements  docu¬ 
ments  are  of  particular  value  to  the  tester 
since  they  form  the  basis  for  the  initial 
identification  of  critical  issues  that  will 
be  addressed  in  the  test  program. 

5.2.3  Operational  Requirements 
Document  (ORD) 

The  ORD  is  first  prepared  for  MS  I  by  the 
user  or  a  user’s  representative  and  is  ap¬ 
proved  by  the  Service  Chief  or  a  desig¬ 
nated  representative.  For  ACAT  ID  pro¬ 
grams,  JROC  will  designate  the  approval 
authority  for  the  ORD.  At  MS  H,  the  up¬ 
dated  ORD  should  contain  thresholds  and 
objectives  for  more  detailed  and  refined 
performance  capabilities  and  character¬ 
istics  based  on  the  results  of  trade-off 
studies  and  testing  conducted  during 
Phase  I.  The  ORD  is  a  translation  of  the 
MNS  into  user  requirements,  and  each 
concept  considered  at  MS  I  will  have  a 
tailored  ORD.  Objectives  and  thresholds 
for  various  system  performance  param¬ 
eters  outlined  in  the  ORD  will  also  be 
found  in  baseline  documents,  the  TEMP 
and  program  specifications.  (Figure  5-2) 
Format  for  the  ORD  can  be  found  in  a 
DoD  5000.2-R  appendix . 


5.2.4  System  Threat  Assessment  (STA) 

An  STA  is  prepared  by  the  DoD  Compo¬ 
nent  Intelligence  Command  or  Agency, 
and  for  ACAT  ID  programs,  and  are  vali¬ 
dated  by  the  Defense  Intelligence  Agency. 
The  STA,  for  Defense  Acquisition  Board 
(DAB)  programs,  will  contain  a  concise 
description  of  the  projected  future  opera¬ 
tional  threat  environment,  the  system- 
specific  threat,  the  reactive  threat  that 
could  affect  program  decisions,  and  when 
appropriate,  the  results  of  interactive 
analysis  obtained  by  the  Service  PM  when 
evaluating  the  program  against  the  threat. 
Threat  projections  start  at  the  initial  oper¬ 
ating  capacity  (IOC)  and  extend  over  the 
following  ten  years.  The  STA  provides 
the  basis  for  the  test  design  of  threat  sce¬ 
narios  and  the  acquisition  of  appropriate 
threat  targets,  equipment,  or  surrogates. 
It  provides  threat  data  for  development 
test  and  evaluation  (DT&E)  and  opera¬ 
tional  test  and  evaluation  (OT&E).  Vul¬ 
nerability  and  lethality  analyses  during 
live  fire  testing  of  ACAT  I  and  II  systems 
are  contingent  on  valid  threat  descrip¬ 
tions.  A  summary  of  the  STA  is  included 
in  part  1  of  the  TEMP. 

5.3  PROGRAM  DECISION 
DOCUMENTATION 

5.3.1  Acquisition  Decision 
Memorandum  (ADM) 

Under  Secretary  of  Defense  for  Acquisi¬ 
tion  and  Technology  (USD(A&T))  deci¬ 
sions  at  major  defense  ACAT  ID  mile¬ 
stones  are  recorded  in  a  document  known 
as  an  ADM.  The  ADM  documents  a 
USD(A&T)  decision  on  an  MNS  at  MS  0 
and  on  the  Aquisition  Program  Baseline 
(APB)  at  Milestones  I,  II  and  III.  In  con¬ 
junction  with  an  ADM  and  its  included 
exit  criteria  for  the  next  phase,  the  APB  is 
a  primary  program  guidance  document 
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Figure  5-2.  Requirements  Definition  Process 


providing  goals /thresholds  for  systems 
performance. 

5.3.3  Analysis  of  Alternatives  (AOA) 

An  AOA  is  normally  prepared  by  a  DoD 
Component  agency  (or  Principal  Staff  As¬ 
sistant  for  ACAT  lA  programs),  other  than 
the  program  management  office,  for  each 
milestone  review  beginning  at  MS  I.  The 
AOA  aids  decision  makers  by  examining 
the  relative  advantages  and  disadvantages 
of  program  alternatives,  shows  the  sensi¬ 
tivity  of  each  alternative  to  possible  changes 
in  key  assumptions,  and  provides  the  ratio¬ 
nale  for  each  option.  The  guidance  in  DoD 
5000.2-R,  part  2,  requires  a  clear  linkage 
between  the  analysis  of  alternatives,  sys¬ 
tem  requirements,  and  system  evaluation 
measures  of  effectiveness. 

The  driving  factor  behind  this  linkage  is  the 
decision  maker's  reluctance  to  accept  mod¬ 
eling  or  simulation  projections  for  system 
performance  in  the  future  without  actual 
test  data  that  validates  AOA  results. 

5.4  PROGRAM  MANAGEMENT 
DOCUMENTATION 

5.4.1  Acquisition  Strategy 

An  event-based  acquisition  strategy  must 
be  formulated  at  the  start  of  a  development 
program  (MS  I).  Event-driven  acquisition 
strategy  explicitly  links  program  decisions 
to  demonstrated  accomplishments  in  de¬ 
velopment,  testing  and  initial  production. 
The  strategy  constitutes  a  broad  set  of  con¬ 
cepts  that  provide  direction  and  control  for 
the  overall  development  and  production 
effort.  The  acquisition  strategy  is  updated  at 
each  MS  decision  point  using  an  Integrated 
ProductTeam  (IPT)  structure  throughout  the 
life  of  a  program.  The  level  of  detail  re¬ 
flected  in  the  acquisition  strategy  can  be 
expected  to  increase  as  a  program  matures. 


The  acquisition  strategy  serves  as  a  tailored 
conceptual  basis  for  formulating  other  pro¬ 
gram  functional  plans  such  as  the  TEMP. 

It  is  important  that  T&E  interests  be  repre¬ 
sented  as  the  acquisition  strategy  is  formu¬ 
lated  because  the  acquisition  strategy 
should: 

•  Provide  an  overview  of  the  T&E 
planned  for  the  program,  ensuring  that 
adequate  T&E  is  conducted  prior  to  the 
production  decision; 

•  Discuss  plans  for  providing  adequate 
quantities  of  test  hardware; 

•  Describe  levels  of  concurrence  and 
combined  development  test/operational 
test  (DT/OT). 

5.4.2  Baseline  Documentation 

The  Acquisition  Program  Baseline  will  ini¬ 
tially  be  developed  by  the  Program  Man¬ 
agement  Office  (PMO)  at  MS  I  and  revised 
for  each  subsequent  milestone.  Baseline 
parameters  represent  the  cost,  schedule  and 
performance  objectives  and  thresholds  for 
the  system  in  a  production  configuration. 
Each  baseline  influences  the  T&E  activities 
in  the  succeeding  phases.  Measures  of  ef¬ 
fectiveness  or  measures  of  performance 
shall  be  used  in  describing  needed  capa¬ 
bilities  early  in  a  program.  Guidance  on  the 
formulation  of  baselines  is  found  in  DoD 
5000.2-R.  Performance  demonstrated  dur¬ 
ing  T&E  of  production  systems  must  meet 
or  exceed  the  thresholds.  The  thresholds 
establish  deviation  limits  (actual  or  antici¬ 
pated  breach  triggers  reports)  for  key  per¬ 
formance  parameters  beyond  which  the 
PM  may  not  trade  off  cost,  schedule  or 
performance  without  authorization  by  the 
Milestone  Decision  Authority  (MDA). 
Baseline  and  test  documentation  must 
reflect  the  same  expectations  for  system 
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performance.  The  total  number  of  perfor¬ 
mance  parameters  shall  be  the  minimum 
number  needed  to  characterize  the  major 
drivers  of  operational  effectiveness  and 
suitability,  schedule,  technical  progress,  and 
cost.  The  performance  parameters  may  not 
completely  define  operational  effectiveness 
or  suitability.  The  MDA  may  add  addi¬ 
tional  performance  parameters  not  vali¬ 
dated  by  the  JROC. 

5.4.3  Acquisition  Logistics  Planning 

Supportability  analyses  are  a  composite  of 
all  support  considerations  necessary  to  en¬ 
sure  the  effective  and  economical  support 
of  a  system  at  all  levels  of  maintenance  for 
its  programmed  life  cycle.  Support  con¬ 
cepts  describe  the  overall  logistic  support 
program  and  include  logistics  require¬ 
ments,  tasks  and  milestones  for  the  current 
and  succeeding  phases  of  the  program.  The 
analyses  serve  as  the  source  document  for 
logistic  support  testing  requirements. 

Guidelines  for  logistic  support  analyses  are 
documented  in  MIL-STD-1388-1A.  This 
standard  identifies  how  T&E  programs 
should  be  planned  to  serve  the  following 
three  logistics  supportability  objectives: 

(1)  Provide  measured  data  for  input  into 
system-level  estimates  of  readiness,  opera¬ 
tional  costs  and  logistics  support  resource 
requirements; 

(2)  Expose  supportability  problems  so 
they  can  be  corrected  prior  to  deployment; 

(3)  Demonstrate  contractor  compliance 
with  quantitative  supportability — related 
design  requirements. 

Development  of  an  effective  T&E  program 
requires  close  coordination  of  efforts  among 
all  system  engineering  disciplines,  espe¬ 
cially  those  involved  in  logistics  support 


analyses.  The  support  analyses  should  be 
drafted  before  MS  I  to  provide  a  skeletal 
framework  for  logistics  support  analysis, 
to  identify  initial  logistics  testing  require¬ 
ments  that  can  be  used  as  input  to  the 
TEMP  and  to  provide  test  feedback  to  sup¬ 
port  Integrated  Logistics  Support  (ILS) 
development.  Test  resources  will  be  lim¬ 
ited  early  in  the  program  because  DoD 
5000.2-R  requires  that  support  resources 
not  be  procured  before  the  weapon  sys¬ 
tem/component  hardware  and  software 
design  stabilizes. 

5.4.4  Specification 

The  system  specification  is  a  document 
used  in  development  and  procurement 
which  describes  the  technical  performance 
requirements  for  items,  materials,  and  ser¬ 
vices  including  the  procedures  by  which  it 
will  be  determined  that  the  requirements 
have  been  met.  Specification  evolves  over 
the  developmental  phases  of  the  program 
with  increasing  levels  of  detail:  system; 
item  performance;  item  detail;  process;  and 
material.  Section  4  of  the  specification  iden¬ 
tifies  what  procedures  (inspection,  demon¬ 
stration,  analysis,  and  test)  will  be  used  to 
verify  the  performance  parameters  listed 
in  section  3.  Further  details  may  be  found  in 
MIL-STI>961D,  Military  Defense  Specifi¬ 
cation  Standard  Practices  (incorporated 
portions  of  MIL-STD-490)  which  is  fully 
exempt  from  the  MIL-STD  waiver  process 
because  it  is  a  "Standard  Practice." 

5.4.5  Work  Breakdown  Structure  (WBS) 

A  program  work  breakdown  structure 
(WBS)  shall  be  established  that  provides  a 
framework  for  program  and  technical  plan¬ 
ning,  cost  estimating,  resource  allocations, 
performance  measurements,  and  status  re¬ 
porting.  Program  offices  shall  tailor  a  pro¬ 
gram  WBS  for  each  program  using  the  guid¬ 
ance  in  MIL-HNBK-881 .  Level  2  of  the  WBS 
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hierarchical  structure  addresses  system 
level  test  and  evaluation  with  sub-levels 
for  DT&E  and  OT&E.  Additionally,  each 
configuration  item  structure  includes  de¬ 
tails  of  the  integration  and  test  require¬ 
ments. 

5.5  TEST  PROGRAM 
DOCUMENTATION 

5.5.1  Test  and  Evaluation 
Master  Plan  (TEMP) 

The  TEMP  is  the  basic  planning  docu¬ 
ment  for  T&E  related  to  a  DoD  system 
acquisition  (Figure  5-3).  It  is  prepared  by 
the  PMO  with  the  operational  test  infor¬ 
mation  provided  by  the  Service  Opera¬ 
tional  Test  Agency.  It  is  used  by  Office  of 
the  Secretary  of  Defense  (OSD)  and  the 
Services  for  planning,  reviewing  and  ap¬ 
proving  T&E  programs  and  provides  the 
basis  and  authority  for  all  other  detailed 
T&E  planning  documents.  The  TEMP 
identifies  critical  technical  parameters 
(CTPs),  characteristics  and  critical  opera¬ 
tional  issues  (COD;  and  it  describes  the 
objectives,  responsibilities,  resources,  and 
schedules  for  all  completed  and  planned 
T&E.  The  TEMP,  in  the  specified  format, 
is  required  by  DoD  5000.2-R  for  AC  AT  I, 
lA,  and  designated  oversight  programs 
(see  appendix  III  for  more  information 
regarding  the  TEMP  format).  Format  is  at 
Service  discretion  for  ACAT  II  and  III 
programs. 

5.5.2  Evaluation  Plan 

Evaluation  planning  is  usually  included 
within  the  test  plan.  Evaluation  planning 
considers  the  evaluation  and  analysis  tech¬ 
niques  that  will  be  required  once  the  test 
data  has  been  collected  and  processed. 
Evaluation  is  linked  closely  to  the  test  de¬ 
sign,  especially  the  statistical  models  on 
which  the  test  design  is  built. 


The  Army  requires  a  system  evaluation 
plan  describing  the  evaluation  being  con¬ 
ducted  by  a  technical  independent  evalua¬ 
tor  or  an  operational  independent  evalua¬ 
tor. 

The  objective  of  the  Army's  emphasis  on 
evaluation  is  to  address  the  issues;  describe 
the  evaluation  of  issues  which  require  data 
from  sources  other  than  test;  state  the  tech¬ 
nical  or  operational  issues  and  criteria;  iden¬ 
tify  data  sources;  state  the  approach  to  the 
independent  evaluation;  specify  the  ana¬ 
lytical  plan  and  identify  program  con¬ 
straints.  (Reference  59) 

Evaluation  plans  are  prepared  for  all  sys¬ 
tems  in  development  by  the  independent 
evaluators  during  concept  exploration  and 
in  coordination  with  the  system  developer. 
The  Army  System  Evaluation  Plan  compli¬ 
ments  the  TEMP  and  is  updated  when  the 
TEMP  is  revised.  It  identifies  each  evalua¬ 
tion  issue  and  the  methodology  to  be  used 
to  assess  it  and  specifies  requirements  for 
exchange  of  information  between  the  de¬ 
velopment/  operational  testers  and  mate¬ 
riel  developers. 

5.5.3  Test  Design 

Test  designers  need  to  ensure  that  the  test  is 
constructed  to  provide  useful  information 
in  all  areas /aspects  that  will  lead  to  an 
assessment  of  the  system  performance.  For 
example,  a  complicated,  even  ingenious, 
test  that  does  not  provide  the  information 
required  by  the  decision  makers  is,  in  many 
respects,  a  failed  endeavor.  Therefore,  part 
of  the  process  of  developing  a  test  concept 
or  test  design  (the  distinction  between  these 
vary  from  organization  to  organization) 
should  be  to  consider  whether  the  test  will 
provide  the  information  required  by  the 
decision  makers.  In  other  words,  "Are  we 
testing  the  right  things  in  the  right  way ..  .and 
are  our  evaluations  meaningful?" 
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Figure  5-3.  Test  Program  Documentation 


The  test  design  is  statistical  and  analytical 
in  nature  and  should  perform  the  follow¬ 
ing  functions: 

(1)  Structure  and  organize  the  approach 
to  testing  in  terms  of  specific  test  objectives; 

(2)  Identify  key  measures  of  effective¬ 
ness  (MOEs)  and  measures  of  performance 
(MOPs); 

(3)  Identify  the  required  data  and  dem¬ 
onstrate  how  the  data  will  be  gathered, 
stored,  analyzed  and  used  to  evaluate 
MOEs; 

(4)  Indicate  what  part  modeling  and 
simulation  will  play  in  meeting  test  objec¬ 
tives; 

(5)  Identify  the  number  and  type  of  test 
events  and  required  resources. 

The  test  design  may  serve  as  a  foundation 
for  the  more-detailed  test  plan  and  speci¬ 
fies  the  test  objectives,  events,  instrumen¬ 
tation,  methodology,  data  requirements, 
data  management  needs  and  analysis  re¬ 
quirements. 

5.5.4  Test  Plan 

The  test  plan  is  the  vehicle  that  translates  a 
test  concept  and  statistical /analytical  test 
design  into  concrete  resources,  procedures 
and  responsibilities.  The  size  and  complex¬ 
ity  of  a  test  program  and  its  associated  test 
plan  are  determined  by  the  nature  of  the 
system  being  tested  and  the  type  of  testing 
that  is  to  be  accomplished.  Some  major 
weapons  systems  may  require  large  num¬ 
bers  of  separate  tests  to  satisfy  test  objec¬ 
tives  and,  thus,  require  a  multi- volume  test 
plan;  other  testing  may  be  well-defined  by 
a  relatively  brief  test  plan.  The  test  plan  also 


provides  a  description  of  the  equipment 
configuration  and  known  limitations  to  the 
scope  of  testing.  The  type  of  information 
typically  included  in  a  test  plan  is  shown  in 
Table  5-1. 

5.5.5  Outline  Test  Plan/  Resources  Plan 

The  Army's  Outline  Test  Plan  (OTP)  and 
Air  Force's  Test  Resources  Plan  (TRP)  are 
essential  test  planning  documents.  They 
are  formal  resource  documents  specifying 
the  resources  required  to  support  the  test. 
Since  the  OTP  or  TRP  provide  the  basis  for 
fiscal  programming  and  coordinating  the 
necessary  resources,  it  is  important  that 
these  documents  be  developed  in  advance 
and  kept  current  to  reflect  maturing  re¬ 
source  requirements  as  the  test  program 
develops.  The  Navy  makes  extensive  use  of 
the  TEMP  to  document  T&E  resource  re¬ 
quirements.  Each  Service  has  periodic  meet¬ 
ings  designed  to  review  resource  require¬ 
ments  and  resolve  problems  with  test  sup¬ 
port. 

5.5.6  Test  Reports 

5.5.6.1  Quick-Look  Reports 

Quick-look  analyses  are  expeditious  analy¬ 
ses  performed  during  testing  using  limited 
amounts  of  the  database.  Such  analyses 
often  are  used  to  assist  in  managing  test 
operations.  Quick-look  reports  are  used 
occasionally  to  inform  higher  authorities  of 
test  results.  Quick-look  reports  may  have 
associated  briefings  that  present  T&E  re¬ 
sults  and  substantiate  conclusions  or  rec¬ 
ommendations.  Quick-look  reports  may  be 
generated  by  the  contractor  or  government 
agency.  They  are  of  particularly  critical 
interest  for  high- visibility  systems  that  may 
be  experiencing  some  development  diffi¬ 
culties.  Techniques  and  formats  should  be 
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Table  5-1.  Sample  Test  Plan  Contents 


PRELIMINARY  PAGES 

i.  TITLE  PAGE 

ii.  ABSTRACT 

iii.  TABLE  OF  CONTENTS 

iv.  TERMS  AND  ABBREVIATIONS 
V.  RELATED  DOCUMENTS* 


*THE  ACTUAL  NUMBER  OF  THESE  PAGES  WILL  BE  DETERMINED  BY  THE  LENGTH 
OF  PRELIMINARY  ELEMENTS  (e.g.,  TABLE  OF  CONTENTS,  TERMS  AND  ABBREVIA¬ 
TIONS,  ETC.). 


MAIN  BODY 

1.  INTRODUCTION 

2.  TEST  PURPOSE  AND  OBJECTIVES 

3.  CONCEPT  OF  TEST  OPERATIONS 

4.  METHOD  OF  ACCOMPLISHMENT 

5.  TEST  SCHEDULE 

6.  TEST  MANAGEMENT  AND  ORGANIZATION 

7.  RESPONSIBILITIES/SUPPORT 

8.  PERSONNEL 

9.  REQUIRED  TEST  REPORTS 

10.  SAFETY 

11.  SECURITY 

12.  INFORMATION 

13.  ENVIRONMENTAL  PROTECTION 


ANNEXES 

A.  TEST  DESIGN 

B.  DATA  REQUIREMENTS 

C.  INSTRUMENTATION  PLAN 

D.  LOGISTICS  SUPPORT  REQUIREMENTS 

E.  RELIABILITY  AND  MAINTAINABILITY  DATA  PLAN 

F.  INTELLIGENCE/THREAT  INFORMATION 
G-Z.  AS  REQUIRED 

1,  2,  3,  ETC.,  DETAILED  TEST  PROCEDURES  (NAME  OF  TEST) 
DISTRIBUTION: 


Source:  “Standard  Procedures  for  USAF  OT&E,"  July  1974. 
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determined  before  the  start  of  testing.  They 
may  be  exercised  dining  pretest  trials. 

5.5.6.2  Final  Test  Report 

The  final  test  report  disseminates  the  test 
information  to  decision  authorities,  pro¬ 
gram  office  staff  and  the  acquisition  com¬ 
munity.  It  provides  a  permanent  record  of 
the  execution  of  the  test  and  its  results.  The 
final  test  report  should  relate  the  test  re¬ 
sults  to  the  critical  issues  and  address  the 
objectives  stated  in  the  test  design  and  test 
plan.  A  final  test  report  may  be  separated 
into  two  sections  —  a  main  section  provid¬ 
ing  the  essential  information  about  test 
methods  and  results,  and  a  second  section 
consisting  of  supporting  appendices  to  pro¬ 
vide  details  and  supplemental  information. 
Generally,  the  following  topics  are  included 
in  the  main  body  of  the  report: 

(1)  Test  purpose 

(2)  Issues  and  objectives 

(3)  Method  of  accomplishment 

(4)  Results  (keyed  to  the  objectives  and 
issues) 

(5)  Discussion,  conclusions  and  recom¬ 
mendations. 

Appendices  of  the  final  test  report  may 
address  the  following  topics: 

(1)  Detailed  test  description 

(2)  Test  environment 

(3)  Test  organization  and  operation 

(4)  Instrumentation 

(5)  Data  collection  and  management 

(6)  Test  data 


(7)  Data  analysis 

(8)  Modeling  and  simulation 

(9)  Reliability,  availability  and  main¬ 
tainability  information 

(10)  Personnel 

(11)  Training 

(12)  Safety 

(13)  Security 

(14)  Funding 

(15)  Asset  Disposition. 

The  final  test  report  may  contain  an  evalu¬ 
ation  and  analysis  of  the  results,  or  the 
evaluation  may  be  issued  separately.  The 
analysis  tells  what  the  results  are,  whereas 
an  evaluation  tells  what  the  results  mean. 
The  evaluation  builds  on  the  analysis  and 
generalizes  from  it,  showing  how  the  re¬ 
sults  apply  outside  the  test  arena.  It  shows 
what  the  implications  of  the  test  are  and 
may  provide  recommendations.  The  evalu¬ 
ation  may  make  use  of  independent  analy¬ 
ses  of  all  or  part  of  the  data;  it  may  employ 
data  from  other  sources  and  may  use  mod¬ 
eling  and  simulation  to  generalize  the  re¬ 
sults  and  extrapolate  to  other  conditions.  In 
the  case  of  the  Army,  a  separate  Indepen¬ 
dent  Evaluation  Report  is  prepared  by  in¬ 
dependent  evaluators  within  Operational 
Test  and  Evaluation  Command  (OPTEC). 

5.6  OTHER  TEST-RELATED 
STATUS  REPORTS 

5.6.1  End  of  Test  Phase  Report 

The  Services  are  required  by  DoD  5000.2-R 
to  submit  to  OSD  T&E  offices  copies  of  their 
formal  detailed  DT&E,  OT&E,  and  live  fire 


5-11 


T&E  reports  that  are  prepared  at  the  end  of 
each  phase  of  testing  for  ACAT  I,  lA,  and 
oversight  programs.  These  reports  will  gen¬ 
erally  be  submitted  45  days  in  advance  of  a 
milestone  or  decision  review. 

5.6.2  Beyond  Low-Rate  Initial 
Production  Report  (BLRIP) 

Before  an  ACAT  I  or  Director,  Operational 
Test  and  Evaluation  (DOT&E)  designated 
program  can  proceed  beyond  (MS  III)  Low 
Rate  Initial  Production  (LRIP),  the  DOT&E 
must  submit  a  BLRIP  report  to  the  Secre¬ 
tary  of  Defense  and  the  Senate  and  House 
of  Representatives  Committees  on  Armed 
Services,  National  Security,  and  Appro¬ 
priations.  This  report  addresses  whether 
the  OT&E  performed  was  adequate  and 
whether  the  OT&E  results  confirm  that  the 


items  or  components  tested  are  effective 
and  suitable  for  use  in  combat  by  typical 
military  users.  The  report  may  include  in¬ 
formation  on  the  results  of  live  fire  T&E  for 
applicable  major  systems. 

5.7  SUMMARY 

A  wide  range  of  documentation  is  avail¬ 
able  to  the  test  manager  and  should  be  used 
to  develop  T&E  programs  that  address  all 
relevant  issues.  The  PM  must  work  to  en¬ 
sure  that  T&E  requirements  are  considered 
at  the  outset  when  the  acquisition  strategy 
is  formulated.  The  PM  must  also  require 
early,  close  coordination  and  a  continuing 
dialogue  among  those  responsible  for  inte¬ 
gration  of  functional  area  planning  and  the 
TEMP. 
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TYPES  OF  TEST  AND  EVALUATION 


6.1  INTRODUCTION 

This  chapter  provides  a  brief  introduction 
to  development  test  and  evaluation  (DT&E) 
and  operational  test  and  evaluation  (OT&E) 
—  two  principal  types  of  test  and  evalua¬ 
tion  (T&E);  it  also  discusses  the  role  of 
qualification  testing  as  a  sub-element  of 
development  testing.  Other  important  types 
of  T&E  are  introduced.  They  include:  multi- 
Service  testing;  joint  T&E;  live  fire  testing; 
nuclear,  chemical  and  biological  testing; 
and  nuclear  hardening  and  survivability 
testing.  As  Figure  6-1  illustrates,  DT&E  and 
OT&E  are  performed  throughout  the  ac¬ 
quisition  process  and  identified  by  nomen¬ 
clature  that  may  change  with  the  phase  of 
the  acquisition  cycle  in  which  they  occur. 

6.2  DEVELOPMENT  TEST 
AND  EVALUATION  (DT&E) 

Development  test  and  evaluation  is  T&E 
conducted  throughout  the  acquisition  pro¬ 
cess  to  assist  in  engineering  design  and 
development  and  to  verify  that  technical 
performance  specifications  have  been  met. 
The  DT&E  is  planned  and  monitored  by 
the  developing  agency  and  is  normally  con¬ 
ducted  by  the  contractor.  However,  the 
development  agency  may  perform  techni¬ 
cal  compliance  tests  before  OT&E.  It  in¬ 
cludes  the  T&E  of  components,  subsystems, 
preplanned  product  improvement  (P^I) 
changes,  hardware/ software  integration 
and  production  qualification  testing.  It  en¬ 
compasses  the  use  of  models,  simulations, 
test  beds,  and  prototypes  or  full-scale  engi¬ 
neering  development  models  of  the  system. 


Development  test  and  evaluation  may  in¬ 
volve  a  wide  degree  of  test  complexity, 
depending  upon  the  type  of  system  or  test 
article  under  development;  e.g.,  tests  of 
electronic  breadboards  or  brassboards, 
components,  subsystems  or  experimental 
prototypes. 

Development  test  and  evaluation  supports 
the  system  design  process  through  an  itera¬ 
tive  Simulate-Test-Evaluate  Process  (STEP) 
that  involves  both  contractor  and  govern¬ 
ment  personnel.  Because  contractor  testing 
plays  a  pivotal  role  in  the  total  test  pro¬ 
gram,  it  is  important  the  contractor  estab¬ 
lishes  an  integrated  test  plan  early  to  en¬ 
sure  that  the  scope  of  the  contractor's  test 
program  satisfies  government  and  contrac¬ 
tor  test  objectives. 

The  program  manager  (PM)  remains  re¬ 
sponsible  for  the  ultimate  success  of  the 
overall  program.  The  PM  and  the  test  spe¬ 
cialists  on  the  PM's  staff  must  foster  an 
environment  that  provides  the  contractor 
with  sufficient  latitude  to  pursue  innova¬ 
tive  solutions  to  technical  problems  and,  at 
the  same  time,  provides  the  data  needed  to 
make  rational  trade-off  decisions  between 
cost,  schedule  and  performance  as  the  pro¬ 
gram  progresses. 

6.2.1  Production  Qualification  Test 
(PQT) 

Qualification  testing  is  a  form  of  develop¬ 
ment  testing  that  verifies  the  design  and 


manufacturing  process.  Production  quali¬ 
fication  tests  are  formal  contractual  tests 
that  confirm  the  integrity  of  the  system 
design  over  the  operational  and  environ¬ 
mental  range  in  the  specification.  These 
tests  usually  use  preproduction  hardware 
fabricated  to  the  proposed  production 
design  specifications  and  drawings.  Such 
tests  include  contractual  reliability  and 
maintainability  demonstration  tests  re¬ 
quired  before  production  release.  Pro¬ 
duction  qualification  T&E  must  be  com¬ 
pleted  before  Milestone  III  lAW  Depart¬ 
ment  of  Defense  (DoD)  5000.2-R. 

Production  qualification  tests  may  be  con¬ 
ducted  on  low  rate  initial  production  items 
to  ensure  the  effectiveness  of  the  manufac¬ 
turing  process,  equipment  and  procedures. 
These  tests  are  conducted  on  each  item  or  a 
sample  lot  taken  at  random  from  the  first 
production  lot  and  are  repeated  if  the  pro¬ 
cess  or  design  is  changed  significantly  or  a 
second  or  alternative  source  is  brought  on 
line.  These  tests  are  also  conducted  against 
contractual  design  and  performance  re¬ 
quirements. 

6.3  OPERATIONAL  TEST 
AND  EVALUATION  (OT&E) 

6.3.1  The  Difference  Between 
Development  and  Operational  Testing 

Air  Force  Manual  55-43,  published  in  June 
1979,  once  contained  the  following  account 
of  the  first  OT&E;  this  anecdote  serves  as  an 
excellent  illustration  of  the  difference  be¬ 
tween  development  and  operational  test¬ 
ing: 

The  test  and  evaluation  of  aircraft  and 
air  weapon  systems  started  with  the 
contract  awarded  to  the  Wright  broth¬ 
ers  in  1908.  This  contract  specified  a 
craft  which  would  lift  two  men  with  a 
total  weight  of  350  pounds,  carry 


enough  fuel  for  a  flight  of  125  miles, 
and  fly  40  miles  per  hour  in  still  air. 
The  contract  also  required  that  testing 
be  conducted  to  assure  this  capability. 

What  we  now  call  development  test 
and  evaluation  (DT&E)  was  satisfied 
when  the  Wright  brothers  (the  devel¬ 
oper)  demonstrated  that  their  airplane 
could  meet  those  first  contract  specifi¬ 
cations.  However,  no  immediate  mili¬ 
tary  mission  had  been  conceived  for 
the  Wright  Flyer.  It  was  shipped  to 
Fort  Sam  Houston,  Texas,  where  Cap¬ 
tain  Benjamin  D.  Foulois,  the  pilot, 
had  orders  to  "teach  himself  to  fly."  He 
had  to  determine  the  airplane's  per¬ 
formance,  how  to  maintain  it,  and  the 
kind  of  organization  that  would  use  it. 
Cavalry  wagon  masters  had  to  be 
trained  as  airplane  mechanics,  and 
Captain  Foulois  was  his  own  instruc¬ 
tor  pilot. 

In  the  process.  Captain  Foulois  sub¬ 
jected  the  Wright  Flyer  to  test  and 
evaluation  under  operational  condi¬ 
tions.  Foulois  soon  discovered  opera¬ 
tional  deficiencies.  For  example,  there 
was  no  seat  on  the  airplane.  During 
hard  landings,  Foulois'  130  pound 
frame  usually  parted  company  from 
the  airplane.  To  correct  the  problem, 
Foulois  bolted  an  iron  tractor  seat  to 
the  airplane.  The  seat  helped,  but 
Foulois  still  toppled  from  his  perch  on 
occasion.  As  a  further  improvement, 
Foulois  looped  his  Sam  Browne  belt 
through  the  seat  and  strapped  himself 
in.  Ever  since  then,  contoured  seats 
and  safety  belts  —  a  product  of  this 
earliest  "operational"  test  and  evalua¬ 
tion  —  have  been  part  of  the  military 
airplane. 

Captain  Foulois’  experience  may  seem  hu¬ 
morous  now,  but  it  dramatically  illustrates 
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the  need  for  operational  testing.  It  also 
shows  that  operational  testing  has  been 
going  on  for  a  long  time. 

As  shown  in  Table  6-1  where  development 
testing  is  focused  on  meeting  detailed  tech¬ 
nical  specifications,  the  operational  test  fo¬ 
cuses  on  the  actual  functioning  of  the  equip¬ 
ment  in  a  realistic  combat  environment  in 
which  the  equipment  must  interact  with 
humans  and  peripheral  equipment.  While 
DT&E  and  OT&E  are  separate  activities 
and  are  conducted  by  different  test  com¬ 
munities,  the  communities  must  interact 
frequently  and  are  generally  complemen¬ 
tary.  The  DT&E  provides  a  view  of  the 
potential  to  reach  technical  objectives,  and 
OT&E  provides  an  assessment  of  the 
system's  potential  to  satisfy  user  require¬ 
ments. 

6.3.2  The  Purpose  of  Operational 
Test  and  Evaluation 

Operational  Test  and  Evaluation  is  defined 
in  Title  10,  U.S.C.  139  and  2399: 

The  field  test,  under  realistic  combat 
conditions,  of  any  item  of  (or  key  com¬ 
ponent  of)  weapons,  equipment,  or 
munitions  for  the  purposes  of  deter¬ 
mining  the  effectiveness  and  suitabil¬ 
ity  of  the  weapons,  equipment,  or 
munitions  for  use  in  combat  by  typical 
military  users;  and  the  evaluation  of 
the  results  of  such  test.  This  term  does 
not  include  an  operational  assessment 
based  exclusively  on  computer  mod¬ 
eling,  simulation,  or  an  analysis  of  sys¬ 
tem  requirements,  engineering  propos¬ 
als,  design  specifications,  or  any  other 
information  contained  in  program 
documents. 

Definitions  of  operational  effectiveness  and 
operational  suitability  are  listed  below: 


Operational  Effectiveness:  The  overall  de¬ 
gree  of  mission  accomplishment  of  a  sys¬ 
tem  when  used  by  representative  person¬ 
nel  in  the  environment  planned  or  expected 
(e.g.  natural,  electronic,  threat  etc.)  for  op¬ 
erational  employment  of  the  system  con¬ 
sidering  organization,  doctrine,  tactics,  sur¬ 
vivability,  vulnerability,  and  threat  (includ¬ 
ing  countermeasures,  initial  nuclear  weap¬ 
ons  effects,  nuclear,  biological  and  chemi¬ 
cal  contamination  (NBCC)  threats). 

Operational  Suitability:  The  degree  to  which 
a  system  can  be  placed  satisfactorily  in  field 
use  with  consideration  given  to  availabil¬ 
ity,  compatibility,  transportability, 
interoperability,  reliability,  wartime  usage 
rates,  maintainability,  safety,  human  fac¬ 
tors,  manpower  supportability,  logistics 
supportability,  natural  environmental  ef¬ 
fects  and  impacts,  documentation  and  train¬ 
ing  requirements. 

In  each  of  the  Services,  operational  testing 
is  conducted  under  the  auspices  of  an  orga¬ 
nization  that  is  independent  of  the  devel¬ 
opment  agency,  in  as  operationally  realis¬ 
tic  environments  as  possible,  with  hostile 
forces  representative  of  the  anticipated 
threat  and  with  typical  users  operating  and 
maintaining  the  system.  In  other  words, 
OT&E  is  conducted  to  ensure  that  new 
systems  meet  the  user's  requirements,  op¬ 
erate  satisfactorily,  and  are  supportable 
under  actual  field  conditions.  The  major 
questions  addressed  in  OT&E  are  shown  in 
Figure  6-2. 

Early  Operational  Assessment,  Operational 
Assessment  (EOA,  OA):  The  OA  normally 
takes  place  during  each  phase  of  the  acqui¬ 
sition  process  prior  to  Milestone  III.They 
are  used  to  provide  an  early  assessment  of 
potential  operational  effectiveness  and  suit¬ 
ability  before  lOT&E.  These  assessments 
attempt  to  project  the  system's  potential  to 
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Table  6-1.  Differences  Between  DT  and  lOT&E 
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Figure  6-2.  Sample  Hierarchy  of  Questions  Addressed  in  Operational  Test  and  Evaluation 


meet  the  user's  requirements.  Assessments 
at  Milestone  n  and  earlier  are  called  EGAs. 

6.3.3  Initial  Operational 
Test  and  Evaluation 

The  OT&E  performed  in  support  of  the 
full-rate  production  decision  (Milestone  HI) 
is  generally  known  as  Initial  Operational 
Test  and  Evaluation  (lOT&E).  The  Navy 
calls  this  eventOPEVAL  (operational evalu¬ 
ation).  The  lOT&E  occurs  in  the  Engineer¬ 
ing  and  Manufacturing  Development 
(EMD)  Phase  and  must  be  completed  be¬ 
fore  Milestone  III.  More  than  one  lOT&E 
may  be  conducted  on  the  system  if  there  are 
system  performance  problems  requiring 
re-test  or  a  need  to  test  in  different  environ¬ 
ments.  The  operational  test  is  conducted 
on  a  production  or  production  representa¬ 
tive  system  using  typical  operational  per¬ 
sonnel  in  a  realistic  combat  scenario. 

6.3.4  Follow-on  Operational 
Test  and  Evaluation 

The  OT&E  performed  after  Milestone  III 
may  be  called  follow-on  operational  test 
and  evaluation  (FOT&E)  and  is  conducted 
during  production,  fielding/deployment, 
operation  and  support.  It,  too,  is  sometimes 
divided  into  two  separate  activities.  Pre¬ 
liminary  FOT&E  is  normally  conducted 
after  the  initial  operational  capability  is 
attained  to  assess  i^U  system  capability.  It 
is  conducted  by  the  OT&E  organization  to 
verify  the  correction  of  deficiencies,  if  re¬ 
quired,  and  to  assess  system  training  and 
logistics  status  not  evaluated  during  lOT &E. 
Subsequent  FOT&E  is  conducted  on  pro¬ 
duction  items  throughout  the  life  of  a  sys¬ 
tem.  The  results  are  used  to  refine  estimates 
of  operational  effectiveness  and  suitabil¬ 
ity;  to  update  training,  tactics,  techniques 
and  doctrine;  and  to  identify  operational 


deficiencies  and  evaluate  modifications. 
This  FOT&E  often  is  conducted  using  the 
operating  command. 

6.4  MULTI-SERVICE  TEST 
AND  EVALUATION 

Multi-Service  test  and  evaluation  is  T&E 
conducted  on  a  system  being  acquired  for 
use  by  more  than  one  Service.  All  affected 
Services  and  their  respective  operational 
test  agencies  participate  in  planning,  con¬ 
ducting,  reporting  and  evaluating  the 
multi-Service  test  program.  One  Service  is 
designated  the  lead  Service  and  is  respon¬ 
sible  for  the  management  of  the  program. 
The  lead  Service  is  charged  with  the  prepa¬ 
ration  and  coordination  of  a  single  report 
that  reflects  the  system's  operational  ef¬ 
fectiveness  and  suitability  for  each  Service. 

The  management  challenge  in  a  joint  ac¬ 
quisition  program  conducting  multi-Ser¬ 
vice  T&E  stems  from  the  fact  that  the  items 
undergoing  testing  will  not  necessarily  be 
used  by  each  of  the  Services  for  identical 
purposes.  Differences  among  the  Services 
usually  exist  in  performance  criteria,  tac¬ 
tics,  doctrine,  configuration  of  armament 
or  electronics  and  the  operating  environ¬ 
ment.  As  a  result,  a  deficiency  or  discrep¬ 
ancy  considered  disqualifying  by  one  Ser¬ 
vice  is  not  necessarily  disqualifying  for  all 
Services.  It  is  incumbent  upon  the  lead 
Service  to  establish  a  discrepancy  report¬ 
ing  system  that  permits  each  participating 
Service  to  document  all  discrepancies 
noted.  At  the  conclusion  of  a  multi-Service 
T&E,  each  participating  OT&E  agency  pre¬ 
pares  an  independent  evaluation  report 
in  its  own  format  and  submits  that  re¬ 
port  through  its  normal  Service  channels. 
The  lead  Service  OT&E  agency  prepares 
the  documentation  that  goes  forward  to 
the  Milestone  Decision  Authority.  This 
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documentation  is  coordinated  with  all  par¬ 
ticipating  OT&E  agencies. 

6.5  JOINT  TEST 
AND  EVALUATION 

Joint  T&E  is  not  the  same  as  multi-Service 
T&E.  Joint  T&E  is  a  specific  program  activ¬ 
ity  sponsored  and  funded  by  an  Office  of 
the  Secretary  of  Defense  (OSD).  Joint  T&E 
programs  are  not  acquisition  oriented;  they 
are  a  means  of  examining  joint-Service  tac¬ 
tics  and  doctrine.  Past  joint-test  programs 
have  been  conducted  to  provide  informa¬ 
tion  required  by  the  Congress,  the  OSD,  the 
commanders  of  the  Unified  Commands 
and  the  Services.  Joint  tests  are  usually 
characterized  as  either  Joint  Development 
T&E  or  Joint  Operational  T&E.  Joint  devel¬ 
opment  T&Es  (Director,  Test,  Systems  En¬ 
gineering  and  Evaluation  charter)  focus  on 
obtaining  information  on  system  require¬ 
ments,  system  performance,  system 
interoperability,  technical  concepts,  tech¬ 
nical  improvements,  improved  testing 
methodologies  or  test  resource  require¬ 
ments. 

Joint  operational  tests  and  evaluations 
(DOTE  charter)  are  conducted  using  actual 
fielded  equipment,  simulators  or  surrogate 
equipment  in  an  exercise  or  operational 
environment  to  obtain  data  pertinent  to 
operational  doctrine,  tactics  and  proce¬ 
dures. 

An  OSD  committee  reviews  candidate 
nominations  for  joint  test  programs  each 
year;  and,  if  a  proposal  is  deemed  appro¬ 
priate  by  the  feasibility  study,  a  lead  Ser¬ 
vice  is  selected  and  tasked  (issued  a  char¬ 
ter)  to  plan  and  execute  the  program  using 
a  test  force  of  participating  ^rvice  person¬ 
nel. 

The  commanders  of  the  four-Service  opera¬ 
tional  test  agencies — the  Army  Operational 


Test  and  Evaluation  Command  (OPTEC), 
the  Navy  Operational  Test  and  Evaluation 
Force  (OPTEVFOR),  the  Air  Force  Opera¬ 
tional  Test  and  Evaluation  Center 
(AFOTEC)  and  the  Marine  Corps  Opera¬ 
tional  Test  and  Evaluation  Activity 
(MCOTEA)  —  have  signed  a  Memoran¬ 
dum  of  Agreement  on  Multi-Service  OT&E 
and  Joint  T&E  (Reference  35)  that  stipu¬ 
lates  how  both  types  of  programs  are  to  be 
managed. 

6.6  LIVE  FIRE  TESTING 

The  Live  Fire  Test  (LFT)  Program  was  man¬ 
dated  by  the  Congress  in  the  National  De¬ 
fense  Authorization  Act  for  Fiscall987  (Pub¬ 
lic  Law  99-661)  passed  in  November  1986. 
Specifically,  this  law  stipulated  that  a  ma¬ 
jor  (Acquisition  Category  (AC AT)  I  and  H) 
program  development  may  not  proceed 
beyond  low  rate  initial  production  until 
realistic  survivability  or  (in  the  case  of  mis¬ 
siles  and  munitions)  lethality  testing  has 
been  completed. 

In  1984,  before  the  passage  of  this  legisla¬ 
tion,  the  OSD  had  chartered  a  joint  test 
program  designed  to  address  similar  ques¬ 
tions  relative  to  systems  already  in  field 
use.  This  program,  the  Joint  LFT,  was  ini¬ 
tially  divided  into  two  distinct  parts:  Ar¬ 
mor/ Anti-armor  and  Aircraft.  The 
program's  objectives  are  to: 

•  Gather  empirical  data  on  the  vulner¬ 
ability  of  existing  U.S.  systems  to  Soviet 
weapons; 

•  Gather  empirical  data  on  the  lethality 
of  existing  U.S.  weapons  against  Soviet 
systems; 

•  Provide  insights  into  the  design  changes 
necessary  to  reduce  vulnerabilities  and  im¬ 
prove  lethalities  of  existing  U.S.  weapon 
systems; 
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•  Calibrate  current  vulnerability  and  le¬ 
thality  models. 

The  legislated  LFT  Program  complements 
the  older  Joint  Live  Fire  (JLF)  Program. 
While  the  JLF  Program  was  designed  to  test 
systems  that  were  fielded  before  being  com¬ 
pletely  tested,  the  spirit  and  intent  of  the 
LFT  legislation  is  to  avoid  the  need  to  play 
"catch-up."  This  program  not  only  requires 
the  Services  to  test  their  weapons  systems 
as  early  as  possible  against  the  expected 
combat  threat  but  also  before  Milestone  III 
to  identify  design  characteristics  that  cause 
undue  combat  damage  or  measure  muni¬ 
tions  lethality.  Remedies  for  deficiencies 
can  entail  required  retrofits,  production 
stoppages  or  other  more  time-consuming 
solutions.  The  essential  feature  of  live  fire 
testing  is  that  appropriate  threat  munitions 
are  fired  against  a  major  U.S.  system  con¬ 
figured  for  combat  to  test  its  vulnerability 
and/or  that  a  major  U.S.  munitions  or  mis¬ 
sile  is  fired  against  a  threat  target  config¬ 
ured  for  combat  to  test  the  lethality  of  the 
munitions  or  missile. 

Live  Fire  Test  and  Evaluation  Guidelines 
were  first  issued  by  the  Deputy  Director, 
T&E  (Live  Fire  Testing)  in  May  1987  to 
supplement  DoD  Test  and  Evaluation  Mas¬ 
ter  Plan  guidelines  (DoD 5000.2-M)  in  areas 
pertaining  to  live  fire  testing  (Reference 
^).  These  guidelines  encompass  all  major 
defense  acquisition  programs  and  define 
LFT  requirements.  In  1994  Public  Law  103- 
355  directed  that  oversight  of  Live  Fire 
Testing  be  moved  within  DoD  to  the  Direc¬ 
tor  of  Operational  Test  and  Evaluation. 
Guidelines  for  this  program  are  now  found 
inDoD5000.2-R. 

6.7  NUCLEAR,  BIOLOGICAL  AND 
CHEMICAL  WEAPONS  TESTING 

The  testing  of  nuclear,  biological  and  chemi¬ 
cal  (NBC)  weapons  is  highly  specialized 


and  regulated.  Program  managers  involved 
in  these  areas  are  advised  to  consult  au¬ 
thorities  within  their  chain  of  command  for 
the  specific  directives,  instructions  and 
regulations  that  apply  to  their  individual 
situations.  Nuclear  weapons  tests  are  di¬ 
vided  into  categories  in  which  the  respon¬ 
sibilities  of  the  Department  of  Energy 
(DOE),  the  Defense  Nuclear  Agency  (DNA) 
and  the  military  Services  are  clearly  as¬ 
signed.  The  DOE  is  responsible  for  nuclear 
warhead  technical  tests;  the  DNA  is  re¬ 
sponsible  for  nuclear  weapons  effects  tests. 
The  Services  are  responsible  for  the  testing 
of  Service-developed  components  of 
nuclear  subsystems.  All  nuclear  tests  are 
conducted  within  the  provisions  of  the  Lim¬ 
ited  Test  Ban  Treaty  that  generally  restricts 
nuclear  detonations  to  the  underground 
environment.  Nuclear  weapons  testing  re¬ 
quires  extensive  coordination  between  Ser¬ 
vice  and  DOE  test  personnel  (Reference 
18). 

Since  the  United  States  signed  and  ratified 
the  Geneva  Protocol  of 1925,  U.S.  policy  has 
been  never  to  be  the  first  to  use  lethal  chemi¬ 
cal  weapons;  it  may,  however,  retaliate  with 
chemical  weapons  if  so  attacked.  With  the 
signing  and  ratification  of  the  1972  Biologi¬ 
cal  and  Toxin  Weapon  Convention,  the 
United  States  formally  adopted  the  posi¬ 
tion  that  it  would  not  employ  biological  or 
toxin  weapons  under  any  circumstances. 
All  such  weapons  were  reported  destroyed 
in  the  early  70s  (Reference  14). 

Regarding  retaliatory  capability  against 
chemical  weapons,  the  Service  Secretaries 
are  responsible  for  ensuring  that  their  or¬ 
ganizations  establish  requirements  and 
determine  the  military  characteristics  of 
chemical  deterrent  items  and  chemical  de¬ 
fense  items.  The  Army  has  been  designated 
the  DoD  executive  agent  for  DoD  chemical 
warfare,  research,  development  and  acqui¬ 
sition  programs  (Reference  14). 
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United  States  policy  on  chemical  warfare 
seeks  to: 

•  Deter  the  use  of  chemical  warfare  weap¬ 
ons  by  other  nations; 

•  Provide  the  capability  to  retaliate  if 
deterrence  fails; 

•  Achieve  the  early  termination  of  chemi¬ 
cal  warfare  at  the  lowest  possible  intensity 
(Reference  14). 

In  addition  to  the  customary  development 
tests  (conducted  to  determine  if  a  weapon 
meets  technical  specifications)  and  opera¬ 
tional  tests  (conducted  to  determine  if  a 
weapon  will  be  useful  in  combat),  chemical 
weapons  testing  involves  two  types  of 
chemical  tests  —  chemical  mixing  and 
biotoxicity.  Chemical-mixing  tests  are  con¬ 
ducted  to  obtain  information  on  the  binary 
chemical  reaction.  Biotoxicity  tests  are  per¬ 
formed  to  assess  the  potency  of  the  agent 
generated.  Chemical  weapons  testing,  of 
necessity,  relies  heavily  on  the  use  of  non¬ 
toxic  stimulants,  since  such  substances  are 
more  economical  and  less  hazardous  and 
open-air  testing  of  live  agents  has  been 
restricted  since  1969  (Reference  14). 

6.8  NUCLEAR  HARDNESS 
AND  SURVIVABILITY  TESTING 

Nuclear  hardness  is  a  quantitative  descrip¬ 
tion  of  the  physical  attributes  of  a  system  or 
component  that  will  allow  it  to  survive  in  a 
given  nuclear  environment.  Nuclear  sur¬ 
vivability  is  the  capability  of  a  system  to 
survive  in  a  nuclear  environment  and  to 
accomplish  a  mission.  Department  of  De¬ 
fense  policy  requires  the  incorporation  of 
nuclear  hardness  and  survivability  features 
in  the  design,  acquisition  and  operation  of 
major  and  nonmajor  systems  that  must 


perform  critical  missions  in  nuclear  con¬ 
flicts.  Nuclear  hardness  levels  must  be  quan¬ 
tified  and  validated  (Reference  15). 

The  T&E  techniques  used  to  assess  nuclear 
hardness  and  survivability  include:  nuclear 
testing,  physical  testing  in  a  simulated  en¬ 
vironment,  modeling,  simulation  and 
analysis.  Although  nuclear  tests  provide  a 
high  degree  of  fidelity  and  valid  results  for 
survivability  evaluation,  they  are  not  prac¬ 
tical  for  most  systems  due  to  cost,  long  lead 
times  and  international  treaty  constraints. 
Underground  testing  is  available  only  on  a 
prioritized  basis  for  critical  equipment  and 
components  and  is  subject  to  a  frequently 
changing  test  schedule.  Physical  testing 
provides  an  opportunity  to  observe  per¬ 
sonnel  and  equipment  in  a  simulated 
nuclear  environment.  Modeling,  simula¬ 
tion  and  analysis  are  particularly  useful  in 
the  early  stages  of  development  to  provide 
early  projections  before  system  hardware 
is  available.  These  methods  are  also  used  to 
furnish  assessments  in  an  area  that,  be¬ 
cause  of  safety  or  testing  limitations,  can¬ 
not  be  directly  observed  through  nuclear  or 
physical  testing. 

6.9  SUMMARY 

Test  and  evaluation  is  a  spectrum  of  tech¬ 
niques  used  to  address  questions  about 
critical  performance  parameters  during 
system  development.  These  questions  may 
involve  several  issues  including:  technical 
(development  testing);  effectiveness,  suit¬ 
ability  and  survivability  (operational  test¬ 
ing);  those  affecting  more  than  one  Service 
(multi-Service  and  joint  testing);  vulner¬ 
ability  and  lethality  (live  fire  testing), 
nuclear  survivability;  or  the  use  of  other 
than  conventional  weapons  (i.e.,  nuclear, 
biological  or  chemical). 
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MODULE 

Developmental 
Test  and  Evaluation 


Material  acquisition  is  an  iterative  process 
of  designing,  building,  testing,  identifying 
deficiencies,  fixing,  retesting  and  repeat¬ 
ing.  Developmental  Test  and  Evaluation 
(DT&E)  is  an  important  aspect  of  this  pro¬ 
cess.  The  DT&E  is  performed  in  the  factory, 
laboratory  and  on  the  proving  ground.  It  is 
conducted  by  subcontractors,  as  they  are 
developing  the  components  and  subassem¬ 
bly;  the  prime  contractor,  as  he /she  as¬ 
sembles  the  components  and  ensures  inte¬ 
gration  of  the  system;  and  by  the  govern¬ 
ment,  to  demonstrate  how  well  the  weapon 
system  meets  its  technical  and  operational 
requirements.  This  module  describes  de¬ 
velopment  testing  and  the  various  types  of 
activities  it  involves.  The  module  also  dis¬ 
cusses  how  development  testing  is  used  to 
support  the  technical  review  process. 


INTRODUCTION  TO  DEVELOPMENT 
TEST  AND  EVALUATION 


7.1  INTRODUCTION 

Development  test  and  evaluation  (DT&E) 
is  the  test  and  evaluation  (T&E)  conducted 
to  demonstrate  that  the  engineering  design 
and  development  process  is  complete.  It  is 
used  by  the  contractor  to  reduce  risk,  vali¬ 
date  and  qualify  the  design,  and  ensure 
that  the  product  is  ready  for  government 
acceptance.  The  DT&E  results  are  evalu¬ 
ated  to  ensure  that  design  risks  have  been 
minimized  and  the  system  will  meet  speci¬ 
fications.  The  results  are  also  used  to  esti¬ 
mate  the  system's  military  utility  when  it  is 
introduced  into  service.  Development  test 
and  evaluation  serves  a  critical  purpose  in 
reducing  the  risks  of  development  by  test¬ 
ing  selected  high-risk  components  or  sub¬ 
systems.  Finally,  DT&E  is  the  government 
developing  agency  tool  used  to  confirm 
that  the  system  performs  as  technically 
specified  and  that  the  system  is  ready  for 
field  testing.  This  chapter  provides  a  gen¬ 
eral  discussion  of  contractor  and  govern¬ 
ment  DT&E  activities,  stresses  the  need  for 
an  integrated  test  program,  describes  some 
special-purpose  development  tests  (DTs) 
and  discusses  several  factors  that  may  in¬ 
fluence  the  extent  and  scope  of  the  DT&E 
program. 

7.2  DT&E  AND  THE  SYSTEM 
ACQUISITION  CYCLE 

As  illustrated  in  Figure  7-1,  DT&E  is  con¬ 
ducted  throughout  the  system  life  cycle. 
Development  test  and  evaluation  may  be¬ 
gin  before  program  initiation  (Milestone  0) 


with  the  evaluation  of  evolving  technology, 
and  it  continues  after  the  system  is  in  the  field. 

7.2.1  DT&E  Prior  to  Program  Initiation 

Prior  to  program  initiation,  modeling,  simu¬ 
lations  and  technology  feasibility  testing  is 
conducted  to  confirm  that  the  technology 
considered  for  the  proposed  weapon  de¬ 
velopment  is  the  most  advanced  available 
and  that  it  is  technically  feasible. 

7.2.2  DT&E  During  Concept 
Exploration 

Development  testing  that  takes  place  dur¬ 
ing  the  Phase  0  is  conducted  by  a  contractor 
or  the  government  to  assist  in  selecting 
preferred  alternative  system  concepts,  tech¬ 
nologies  and  designs.  The  testing  conducted 
depends  on  the  state  of  development  of  the 
test  article's  design.  Government  test  evalu¬ 
ators  participate  in  this  testing  because  in¬ 
formation  obtained  can  be  used  to  support 
the  Systems  Requirements  Review.  The  in¬ 
formation  obtained  from  these  tests  may 
also  be  used  to  support  a  program  start 
decision  by  the  Services  or  the  Office  of  the 
Secretary  of  Defense  (OSD). 

7.2.3  DT&E  During  Program  Definition 
and  Risk  Reduction 

Development  testing  conducted  during 
the  Phase  I  is  used  to  demonstrate  that: 
technical  risk  areas  have  been  identified 
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and  can  be  reduced  to  acceptable  levels;  the 
best  technical  approach  can  be  identified; 
and,  from  this  point  on,  engineering  efforts 
will  be  required  rather  than  experimental 
efforts.  It  supports  the  Milestone  II  deci¬ 
sion  that  considers  entry  into  Engineering 
and  Manufacturing  Development  (EMD) 
and,  as  appropriate,  low  rate  initial  pro¬ 
duction  (LRIP).  This  DT&E  includes  con¬ 
tractor/  government  integrated  testing,  en¬ 
gineering  design  testing  and  advanced  de¬ 
velopment  verification  testing. 

Development  testing  during  this  period  is 
most  often  conducted  at  the  contractor's 
facility.  It  is  conducted  on  components, 
subsystems,  brassboard  configurations  or 
advanced  development  prototypes  to 
evaluate  the  potential  application  of  tech¬ 
nology  and  related  design  approaches  be¬ 
fore  EMD.  Component  interface  problems 
and  equipment  performance  capabilities 
are  evaluated.  The  use  of  properly  vali¬ 
dated  analysis,  modeling  and  simulation  is 
encouraged,  especially  during  the  early 
phases  to  assess  those  areas  that,  for  safety 
or  testing  capability  limitations,  cannot  be 
observed  directly  through  testing.  Mod¬ 
els  and  simulations  can  provide  early  pro¬ 
jections  of  systems  performance,  effec¬ 
tiveness  and  suitability  and  can  reduce 
testing  costs.  This  T&E  also  may  include 
initial  environmental  assessments. 

Army  testing  of  the  Advanced  Attack  Heli¬ 
copter  (AAH)  provides  an  example  of  the 
type  of  activities  that  occur  during  DTs. 
The  early  DT&E  of  the  AAH  was  conducted 
by  the  Army  Engineering  Flight  Activity. 
The  test  was  conducted  in  conjunction  with 
an  Early  Operational  Test,  and  candidate 
designs  were  flown  more  than  90  hours  to 
evaluate  flight  handling  qualities  and  air¬ 
craft  performance.  This  test  also  included  the 
firing  of  the  30  millimeter  cannon  and  the 
2.75-inchrockets.  Reliability,  availability  and 
maintainability  (RAM)  data  were  obtained 


throughout  the  test  program;  and  these 
data,  along  with  RAM  data  provided  from 
early  contractor  testing,  became  a  part  of 
the  system's  RAM  database.  After  evaluat¬ 
ing  the  results,  the  Army  selected  a  contrac¬ 
tor  to  proceed  with  the  next  development 
phase  of  the  AAH. 

7.2.4  DT&E  During  Engineering 
and  Manufacturing  Development 

Development  test  and  evaluation  con¬ 
ducted  during  the  Phase  11  provides  the 
final  technical  data  for  determining  a 
system's  readiness  to  transition  into  either 
low  rate  initial  production  (LRIP)  or  full- 
rate  production.  It  is  conducted  using  ad¬ 
vanced  engineering  development  models 
and  is  characterized  by  engineering  and 
scientific  approaches  under  controlled  con¬ 
ditions.  The  qualification  testing  provides 
quantitative  and  qualitative  data  for  use  in 
the  system's  evaluation.  The  evaluation 
results  are  used  by  the  development  com¬ 
munity  and  are  also  provided  to  Service 
and  OSD  decision  authorities.  These  tests 
measure  technical  performance  including: 
effectiveness,  rehability,  availability,  main¬ 
tainability,  compatibility,  interoperability, 
safety  and  supportability .  They  include  tests 
of  human  engineering  and  technical  as¬ 
pects  of  the  system.  Demonstrations  of 
whether  engineering  is  reasonably  com¬ 
plete  and  if  solutions  to  all  significant  de¬ 
sign  problems  are  in  hand  are  also  included. 

7.2.4.1  Certification  For  lOT&E 

Development  test  and  evaluation  may  be 
conducted  on  engineering  development 
models  or  LRIP  articles  as  a  prelude  to 
certif5dng  the  system  ready  for  Initial  Op¬ 
erational  Test  and  Evaluation  (lOT&E). 
Each  Service  has  different  and  specific  pro¬ 
cesses  incorporated  in  the  certification  for 
lOT&E  documentation.  The  Navy  conducts 
additional  DT&E  for  certification  called 
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TECHEVAL  (technical  evaluation).  This 
is  a  DT&E  event  that  is  conducted  in  a 
more  operationally  realistic  test  environ¬ 
ment.  The  Air  Force  has  developed  a  guide 
with  a  structured  process  using  templates 
to  assist  the  program  manager  (PM)  in 
assessing  the  program's  readiness  for 
lOT&E. 

7.2.4.2  Example  of  EMD  Testing 

As  an  example  of  testing  done  during  the 
EMD  Phase,  the  Army  AAH  was  flown  in 
a  series  of  engineering  design  tests  (EDTs). 
The  EDT-1,  -2  and  -4  were  flown  at  the 
contractor's  facility.  (The  EDT-3  require¬ 
ment  was  deleted  during  program  re¬ 
structuring.)  The  objectives  of  these  flight 
tests  were  to  evaluate  the  handling  char¬ 
acteristics  of  the  aircraft,  check  signifi¬ 
cant  performance  parameters  and  con¬ 
firm  the  correction  of  deficiencies  noted 
during  earlier  testing.  The  EDT-5  was 
conducted  at  an  Army  test  facility,  Yuma 
Proving  Ground.  The  objectives  of  this 
test  were  the  same  as  earlier  EDTs;  how¬ 
ever,  the  testers  were  required  to  ensure 
that  all  discrepancies  were  resolved  be¬ 
fore  the  aircraft  entered  operational  test¬ 
ing.  During  the  EDTs,  operational  test 
personnel  were  completing  operational 
test  design,  bringing  together  test  re¬ 
sources  and  observing  the  DT&E  tests. 
Additionally,  operational  test  (OT)  per¬ 
sonnel  were  compiling  test  data,  such  as 
the  system  contractor's  test  results,  from 
other  sources.  The  evolving  DT  results 
and  contractor  data  were  made  available 
to  the  Critical  Design  Review  members  to 
ensure  that  each  configuration  item  de¬ 
sign  was  essentially  completed.  The  Army 
conducted  a  Physical  Configuration  Au¬ 
dit  (PCA)  to  provide  a  technical  examina¬ 
tion  to  verify  that  each  item  "as  built" 
conformed  to  the  technical  documenta¬ 
tion  defining  that  item. 


7.2.5  DT&E  During  Production, 

Fielding/Deployment 

and  Operational  Support 

Development  testing  may  be  necessary 
after  the  full-rate  production  decision  is 
made  at  Milestone  III.  This  testing  is  nor¬ 
mally  tailored  to  verify  correction  of  iden¬ 
tified  design  problems  and  demonstrate 
the  system  modification's  readiness  for 
production.  This  testing  is  conducted 
under  controlled  conditions  and  provides 
quantitative  and  qualitative  data.  This 
testing  is  conducted  on  production  items 
delivered  from  either  the  pilot  or  initial 
production  runs.  To  ensure  that  the  items 
are  produced  according  to  contract  speci¬ 
fication,  limited  quantity  production  sam¬ 
pling  processes  are  used.  This  testing  de¬ 
termines  whether  the  system  has  success¬ 
fully  transitioned  from  engineering  de¬ 
velopment  prototype  to  production  and 
whether  it  meets  design  specifications. 

The  DT,  which  occurs  soon  after  the  ini¬ 
tial  operating  capability  or  initial  deploy¬ 
ment,  assesses  the  deployed  system's 
operational  readiness  and  supportabil- 
ity.  It  ensures  that  all  deficiencies  noted 
during  previous  testing  have  been  cor¬ 
rected,  evaluates  proposed  product  im¬ 
provements  and  block  upgrades,  and  en¬ 
sures  that  integrated  logistics  support  is 
complete.  It  also  evaluates  the  resources 
on  hand  and  determines  if  the  plans  to 
ensure  operational  phase  readiness  and 
support  objectives  are  sufficient  to  main¬ 
tain  the  system  for  the  remainder  of  its 
acquisition  life  cycle.  For  mature  systems, 
DT&E  is  performed  to  assist  in  modify¬ 
ing  the  system  to  help  meet  new  threats, 
add  new  technologies,  or  aid  in  extend¬ 
ing  service  life. 

Once  a  system  approaches  the  end  of  its 
usefulness,  the  DT  conducted  is  concerned 
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with  the  monitoring  of  a  system's  current 
state  of  operational  effectiveness,  suitabil¬ 
ity  and  readiness  to  determine  whether 
major  upgrades  are  necessary  or  deficien¬ 
cies  warrant  consideration  of  a  new  system 
replacement.  Tests  are  normally  conducted 
by  the  operational  testing  community;  how¬ 
ever,  the  DT&E  community  may  be  re¬ 
quired  to  assess  the  technical  aspects  of  the 
system. 

7.3  DT&E  RESPONSIBILITIES 

As  illustrated  in  Figure  7-1,  the  primary 
participants  in  testing  are  the  prime  con¬ 
tractor,  subcontractor.  Service  materiel  de¬ 
veloper  or  developing  agency  and  the  op¬ 
erational  test  and  evaluation  (OT&E) 
agency.  In  some  Services,  there  are  also 
independent  evaluation  organizations  that 
assist  the  testing  organization  in  designing 
and  evaluating  developrhent  tests.  As  the 
figure  shows,  system  development  testing 
is  performed  principally  by  contractors 
during  the  early  development  stages  of  the 
acquisition  cycle  and  by  government  test/ 
evaluation  organizations  during  the  later 
phases. 

Army  testing  of  the  AAH  illustrates  the 
type  of  DT  performed  by  contractors  and 
the  relationship  of  this  type  of  testing  to 
government  DT&E  activities.  During  the 
contractor  competitive  Phase  I  testing  of 
the  Army  AAH,  prime  contractor  and  sub¬ 
contractor  testing  included  design  support 
tests,  testing  of  individual  components,  es¬ 
tablishing  fatigue  limits,  and  bench  testing 
of  dynamic  components  to  demonstrate 
sufficient  structural  integrity  to  conduct 
the  Army  competitive  flight  test  program. 
Complete  dynamic  system  testing  was 
conducted  utilizing  ground  test  vehicles. 
Besides  supporting  the  contractor's  de¬ 
velopment  effort,  these  tests  provided 


information  for  the  Army  technical  review 
process  as  the  systems,  preliminary  and 
critical  design  reviews  were  conducted.  Fol¬ 
lowing  successful  completion  of  the  ground 
test  vehicle  qualification  testing,  first  flights 
were  conducted  on  the  two  types  of  com¬ 
peting  helicopters.  Each  aircraft  was  being 
flown  300  hours  before  delivery  of  two  of 
each  competing  aircraft  to  the  Army.  The 
contractor  flight  testing  was  oriented  to¬ 
ward  flight-envelope  development,  dem¬ 
onstration  of  structural  integrity,  and  evalu¬ 
ation  and  verification  of  aircraft  flight  han¬ 
dling  qualities.  Some  weapons  system  test¬ 
ing  was  conducted  during  this  phase.  Gov¬ 
ernment  testers  used  much  of  the 
contractor's  testing  data  to  develop  the  test 
data  matrices  as  part  of  the  government's 
DT  and  OT  planning  efforts.  The  use  of 
contractor  test  data  reduced  the  testing  re¬ 
quired  by  the  government  and  added  va¬ 
lidity  to  the  systems  already  tested  and 
data  received  from  other  sources. 

7.3.1  Contractor  Testing 

Materiel  development,  testing  and  evalua¬ 
tion  are  an  iterative  process  in  which  a 
contractor  designs  hardware  and  software, 
evaluates  performance,  makes  changes  as 
necessary,  and  retests  for  performance  and 
technical  compliance.  Contractor  testing 
plays  a  primary  role  in  the  total  test  pro¬ 
gram,  and  the  results  of  contractor  tests  are 
useful  to  the  government  evaluator  in  sup- 
p>orting  government  test  objectives.  It  is 
important  that  government  evaluators,  as 
appropriate,  oversee  contractor  system  tests 
and  use  test  data  from  them  to  address 
government  testing  issues.  It  is  not  uncom¬ 
mon  for  contractor  testing  to  be  conducted 
at  government  test  facilities,  since  contrac¬ 
tors  often  do  not  have  the  required  special¬ 
ized  facilities  (e.g.,  for  testing  hazardous 
components  or  for  missile  flight  tests).  This 
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enables  government  evaluators  to  monitor 
the  tests  more  readily  and  increases  gov¬ 
ernment  confidence  in  the  test  results. 

Normally,  a  Request  For  Proposal  (RFP) 
requires  that  the  winning  contractor  sub¬ 
mit  an  Integrated  Engineering  Design  Test 
Plan  within  a  short  period  after  contract 
initiation  for  coordination  with  government 
test  agencies  and  approval.  This  test  plan 
should  include  testing  required  by  the  State¬ 
ment  of  Work  (SOW),  specifications,  and 
testing  expected  as  part  of  the  engineering 
development  and  integration  process. 
When  approved,  the  contractor's  test  pro¬ 
gram  automatically  becomes  part  of  the 
development  agency's  Integrated  Test  Plan. 

If  the  contractor  has  misinterpreted  the 
RFP  requirements  and  the  Integrated  Engi¬ 
neering  Design  Test  Plan  does  not  satisfy 
government  test  objectives,  the  iterative 
process  of  amending  the  contractor's  test 
program  begins.  This  iterative  process  must 
be  accomplished  within  limited  bounds  so 
the  contractor  can  meet  the  test  objectives 
without  significant  effects  on  contract  cost, 
schedule  or  scope. 

7.3.2  Government  Testing 

Government  testing  is  performed  to:  dem¬ 
onstrate  how  well  the  materiel  system  meets 
its  technical  compliance  requirements,  pro¬ 
vide  data  to  assess  developmental  risk  for 
decision-making,  verify  that  the  technical 
and  support  problems  identified  in  previ¬ 
ous  testing  have  been  corrected,  and  en¬ 
sure  that  all  critical  issues  to  be  resolved  by 
testing  have  been  adequately  considered. 
All  previous  testing,  from  the  contractor's 
bench  testing  through  development 
agency  testing  of  representative  proto¬ 
types,  is  considered  during  government 
evaluation. 

Government  materiel  development  organi¬ 
zations  include  major  materiel  acquisition 


commands  and,  in  some  cases,  operational 
commands.  The  materiel  acquisition  com¬ 
mands  have  T&E  organizations  that  con¬ 
duct  government  development  testing. 
In  addition  to  monitoring  and  participat¬ 
ing  in  contractor  testing,  these  organiza¬ 
tions  conduct  development  testing  on  se¬ 
lected  high-concern  areas  to  evaluate  the 
adequacy  of  systems  engineering,  design, 
development  and  performance  to  specifi¬ 
cations.  The  program  management  office 
(PMO)  must  be  involved  in  all  stages  of 
testing  that  these  organizations  perform. 

In  turn,  the  materiel  development/ test  and 
evaluation  agencies  conduct  T&E  of  the 
systems  in  the  development  stage  to  ensure 
they  meet  technical  and  operational  re¬ 
quirements.  These  organizations  operate 
government  proving  grounds,  test  facili¬ 
ties  and  labs;  and  they  must  be  responsive 
to  the  needs  of  the  PM  by  providing  test 
facilities,  personnel  and  data  acquisition 
services,  as  required. 

7.4  TEST  PROGRAM 
INTEGRATION 

During  the  development  of  a  weapon  sys¬ 
tem,  there  are  a  number  of  tests  conducted 
by  subcontractors,  the  prime  contractor  and 
the  government.  To  ensure  these  tests  are 
properly  time-phased,  that  adequate  re¬ 
sources  are  available,  and  to  minimize  un¬ 
necessary  testing,  a  coordinated  test  pro¬ 
gram  must  be  developed  and  followed. 
The  Test  and  Evaluation  Master  Plan 
(TEMP)  normally  does  not  provide  a  suffi¬ 
cient  level  of  detail  concerning  contractor 
or  subcontractor  tests.  A  contractor  or  PMO 
Integrated  Test  Plan  (FTP)  must  also  be 
developed  to  describe  these  tests.  The  PM 
is  responsible  for  coordinating  the  total 
T&E  program.  The  PM  performs  this  task 
with  the  assistance  of  the  T&E  IPT  whose 
members  are  assembled  from  development 
agency,  user,  technical  and  operational 
T&E,  logistics,  and  training  organizations. 
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The  PM  must  remain  active  in  all  aspects  of 
testing  including  planning,  funding, 
resourcing,  execution  and  reporting.  The 
PM  plays  an  important  role  as  the  interface 
between  the  contractor  and  the  govern¬ 
ment  testing  community.  Recent  emphasis 
on  early  T&E  highlights  a  need  for  early 
government  tester  involvement  in  contrac¬ 
tor  testing.  For  example,  during  develop¬ 
ment  of  the  AAH  test,  it  was  found  that 
having  program  management  personnel 
on  the  test  sites  improved  test  continuity, 
facilitated  the  flow  of  spare  and  repair  parts, 
provided  a  method  of  monitoring  contrac¬ 
tor  performance,  and  kept  the  Service  head¬ 
quarters  informed  with  timely  status  re¬ 
ports. 

7.4.1  Integrated  Test  Plan 

The  ITP  is  used  to  record  the  individual  test 
plans  for  the  subcontractor,  prime  contrac¬ 
tor  and  government.  The  prime  contractor 
should  be  contractually  responsible  for  the 
preparation  and  updating  of  the  HP,  and 
the  contractor  and  Service-developing 
agency  should  ensure  that  it  remains  cur¬ 
rent.  The  ITP  includes  all  developmental 
tests  that  will  be  performed  by  the  prime 
contractor  and  the  subcontractors  at  both 
the  system  and  subsystem  levels.  It  is  a 
detailed,  working-level  document  that  as¬ 
sists  in  identifying  risk  as  well  as  duplica¬ 
tive  or  missing  test  activities.  A  well-main¬ 
tained  ITP  facilitates  the  most  efficient  use 
of  test  resources. 

7.4.2  Single  Integrated  Test  Policy 

Most  Services  have  adopted  a  single  inte¬ 
grated  contractor/government  test  policy, 
thereby  reducing  much  of  the  government 
testing  requirements.  This  policy  stresses 
independent  government  evaluation  and 
permits  an  evaluator  to  monitor  contrac¬ 
tor  and  government  test  programs  and 
evaluate  the  system  from  an  independent 


perspective.  The  policy  stresses  the  use  of 
data  from  all  sources  for  system  evalua¬ 
tion. 

7.5  AREAS  OF  DT&E  FOCUS 

7.5.1  Life  Testing 

Life  testing  is  performed  to  assess  the  ef¬ 
fects  of  long-term  exposure  to  various  por¬ 
tions  of  the  anticipated  environment.  These 
tests  are  used  to  ensure  the  system  will  not 
fail  prematurely  due  to  metal  fatigue,  com¬ 
ponent  aging  or  other  problems  caused  by 
long-term  exposure  to  environmental  ef¬ 
fects.  It  is  important  that  the  requirements 
for  life  testing  are  identified  early  and  inte¬ 
grated  into  the  system  test  plan.  Life  tests 
are  time-consuming  and  costly;  therefore, 
life  testing  requirements  and  life  character¬ 
istics  must  be  carefully  analyzed  concur¬ 
rent  with  the  initial  test  design.  Aging  fail¬ 
ure  data  must  be  collected  early  and  ana¬ 
lyzed  throughout  the  testing  cycle.  If  life 
characteristics  are  ignored  until  results  of 
the  test  are  available,  extensive  redesign 
and  project  delays  may  be  required.  Accel¬ 
erated  life  testing  techniques  are  available 
and  may  be  used  whenever  applicable. 

7.5.2  Design  Evaluation/ 

Verification  Testing 

Design  evaluation  and  verification  testing 
is  conducted  by  the  contractor  and/ or  the 
development  agency  with  the  primary  ob¬ 
jective  of  influencing  system  design.  De¬ 
sign  evaluation  is  fully  integrated  into  the 
development  test  cycle;  and  its  purposes 
are  to; 

(1)  Determine  if  critical  system  technical 
characteristics  are  achievable; 

(2)  Provide  data  for  refining  and  making 
the  hardware  more  rugged  to  comply  with 
technical  specification  requirements; 
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(3)  Eliminate  as  many  technical  and  de¬ 
sign  risks  as  possible  or  to  determine  the 
extent  to  which  they  are  manageable; 

(4)  Provide  for  evolution  of  design  and 
verification  of  the  adequacy  of  design 
changes; 

(5)  Provide  information  in  support  of 
development  efforts; 

(6)  Ensure  components,  subsystems  and 
systems  are  adequately  developed  before 
beginning  operational  tests. 

7.5.3  Design 
Limit  Testing 

Design  limit  tests  are  integrated  into  the 
test  program  to  ensure  the  system  will  pro¬ 
vide  adequate  performance  when  operated 
at  outer  performance  limits  and  when  ex¬ 
posed  to  environmental  conditions  ex¬ 
pected  at  the  extremes  of  the  operating 
envelope.  The  tests  are  based  on  mission 
profile  data.  Care  must  be  taken  to  ensure 
all  systems  and  subsystems  are  exposed  to 
the  worst-case  environments,  with  adjust¬ 
ments  made  because  of  stress  amplification 
factors  and  cooling  problems.  Care  must 
also  be  taken  to  ensure  that  the  system  is 
not  operated  beyond  the  specified  design 
limits;  for  example,  an  aircraft  component 
may  have  to  be  tested  at  temperature  ex¬ 
tremes  from  an  Arctic  environment  to  a 
desert  environment. 

7.5.4  Reliability  Development 
Testing  (RDT) 

Reliability  development  testing  (RDT)  or 
rehability  growth  testing  (RGT)  is  a  planned 
test,  analyze,  fix  and  test  (TAFT)  process  in 
which  development  items  are  tested  under 
actual  or  simulated  mission-profile  envi¬ 
ronments  to  disclose  design  deficiencies 
and  to  provide  engineering  information  on 


failure  modes  and  mechanisms.  The  pur¬ 
pose  of  RDT  is  to  provide  a  basis  for  early 
incorporation  of  corrective  actions  and  veri¬ 
fication  of  their  effectiveness  in  improving 
the  reliability  of  equipment.  Reliability 
development  testing  is  conducted  under 
controlled  conditions  with  simulated  op¬ 
erational  mission  and  environmental  pro¬ 
files  to  determine  design  and  manufac¬ 
turing  process  weaknesses.  The  RDT  pro¬ 
cess  emphasizes  reliability  growth  rather 
than  a  numerical  measurement.  Reliabil¬ 
ity  growth  during  RDT  is  the  result  of  an 
iterative  design  process  because,  as  the 
failures  occur,  the  problems  are  identi¬ 
fied,  solutions  proposed,  the  redesign  is 
accomplished,  and  the  RDT  continues.  A 
substantial  reliability  growth  TAFT  test¬ 
ing  effort  was  conducted  on  the  F-18 
DT&E  for  selected  avionics  and  mechani¬ 
cal  systems.  Although  the  TAFT  effort 
added  $100  million  to  the  Research,  De¬ 
velopment,  Test  and  Evaluation  (RDT&E) 
Program,  it  is  estimated  that  many  times 
that  amount  will  be  saved  through  lower 
operational  and  maintenance  costs 
throughout  the  system's  life. 

7.5.5  Reliability,  Availability 
and  Maintainability  (RAM) 

The  RAM  requirements  are  assessed  dur¬ 
ing  all  contractor  and  government  testing. 
The  data  are  collected  from  each  test  event 
and  placed  in  a  RAM  database,  which  is 
managed  by  the  development  agency.  Con¬ 
tractor  and  government  development  tests 
provide  a  measure  of  the  system's  common 
RAM  performance  against  stated  specifi¬ 
cations  in  a  controlled  environment.  The 
primary  emphasis  of  RAM  data  collection 
during  the  DT  is  to  provide  an  assessment 
of  the  system  RAM  parameters  growth  and 
a  basis  for  assessing  the  consequences  of 
any  differences  anticipated  during  field 
operations.  Early  projections  of  RAM  are 
important  to  logistics  support  planners. 
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The  test  data  facilities  determination  of 
spares  quantities,  maintenance  procedures 
and  support  equipment. 

7.6  SYSTEM  DESIGN 
FORTESTING 

Built-in  test  (BIT),  built-in-test  equipment 
(BITE)  and  automated  test  equipment  (ATE) 
are  major  areas  that  must  be  considered 
from  the  start  of  the  design  effort.  Design 
for  testing  (Figure  7-2)  addresses  the  need 
to:  (1)  collect  data  during  the  development 
process  concerning  particular  performance 
characteristics;  (2)  enable  efficient  and  eco¬ 
nomical  production  by  providing  ready 
access  to,  and  measurement  of,  appropri¬ 
ate  acceptance  parameters;  and  (3)  enable 
rapid  and  accurate  assessment  of  the  status 
of  the  product  to  the  lowest  repairable  ele¬ 
ment  when  deployed.  Many  hardware  sys¬ 
tems  have  testing  circuits  designed  and 
built-in.  This  early  planning  by  design  en¬ 
gineers  allows  easy  testing  for  fault  isola¬ 
tion  of  circuits,  both  in  system  develop¬ 
ment  phases  and  during  operational  test¬ 
ing  and  deployment.  There  are  computer 
chips  in  which  more  than  half  of  the  circuits 
are  for  test/circuit  check  functions.  This 
type  of  circuit  design  requires  early  plan¬ 
ning  by  the  PM  to  ensure  the  RFP  require¬ 
ments  include  the  requirement  for  de- 
signed/BIT  capability.  Evaluation  of  these 
BIT/BITE/ ATE  systems  must  be  included 
in  the  test  program. 

7.7  IMPACT  OF 
WARRANTIES  ON  T&E 

A  warranty  or  guarantee  is  a  commitment 
provided  by  a  supplier  to  deliver  a  product 
that  meets  specified  standards  for  a  speci¬ 
fied  time.  With  a  properly  structured  war¬ 
ranty,  the  contractor  must  meet  technical 
and  operational  requirements.  If  the  prod¬ 
uct  should  fail  during  that  warranty  pe¬ 
riod,  the  contractor  must  replace  or  make 


repairs  at  no  additional  cost  to  the  govern¬ 
ment.  The  Defense  Appropriations  Act  of 
1984  requires  warranties  or  guarantees  on 
all  weapon  systems  procurement.  This  act 
makes  warranties  a  standard  item  on  most 
fixed-price  production  contracts.  Incentives 
are  the  main  thrust  of  warranties,  and  the 
government  will  perform  a  reliability  dem¬ 
onstration  test  on  the  system  to  determine 
these  incentives.  Although  warranties  have 
favorable  advantages  to  the  government 
during  the  early  years  of  the  contract,  war¬ 
ranties  do  not  affect  the  types  of  testing 
performed  to  ensure  the  system  meets  tech¬ 
nical  specifications  and  satisfies  operational 
effectiveness  and  suitability  requirements. 
Warranties  do,  however,  affect  the  amount 
of  testing  required  to  establish  reliability. 
Because  the  standard  item  is  warranted, 
less  emphasis  on  that  portion  of  the  item 
can  allow  for  additional  emphasis  on  other 
aspects  of  the  item  not  covered  under  the 
warranty.  Further,  the  government  may 
tend  to  have  more  confidence  in  contractor 
test  results  and  may  be  able,  therefore,  to 
avoid  some  duplication  of  test  effort.  The 
warranty  essentially  shifts  the  burden  of 
risk  from  the  government  to  the  contractor. 
Warranties  can  significantly  increase  the 
price  of  the  contract,  especially  if  high-cost 
components  are  involved. 

7.8  DT&E  OF  LIMITED 
PROCUREMENT  QUANTITY 
PROGRAMS 

Programs  that  involve  the  procurement  of 
relatively  few  items,  such  as  satellites,  some 
large  missiles,  and  unique  intelligence 
equipment,  typically  over  an  extended  pe¬ 
riod,  are  normally  subjected  to  modified 
DT&E.  Occasionally,  a  unique  test  approach 
that  deviates  from  the  standard  timing  and 
reporting  schedule  will  be  used .  The  DT &E 
principle  of  iterative  testing  starting  with 
components,  subsystems,  prototypes  and 
first-production  models  of  the  system  is 
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Figure  7-2.  Design  for  Testing:  Procedures 


normally  applied  to  limited  procure¬ 
ments.  It  is  important  that  DT&E  and 
OT&E  organizations  work  together  to 
ensure  that  integrated  T&E  plans  are 
adapted /tailored  to  the  overall  acquisi¬ 
tion  strategy. 

7.9  SUMMARY 

Development  test  and  evaluation  is  an 
iterative  process  of  designing,  building. 


testing,  identifying  deficiencies,  fixing, 
retesting  and  repeating.  It  is  performed 
in  the  factory,  laboratory  and  on  the  prov¬ 
ing  ground  by  the  contractors  and  the 
government.  Contractor  and  government 
testing  is  combined  into  one  integrated 
test  program  and  conducted  to  determine 
if  the  performance  requirements  have 
been  met  and  to  provide  data  to  the  deci¬ 
sion  authority. 
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DT&E  SUPPORT  OF  TECHNICAL  REVIEWS 
AND  MILESTONE  DECISIONS 


8.1  INTRODUCTION 

Throughout  the  acquisition  process,  devel¬ 
opment  test  and  evaluation  (DT&E)  is  ori¬ 
ented  toward  the  demonstration  of  specifi¬ 
cations  showing  the  completeness  and  ad¬ 
equacy  of  systems  engineering,  design, 
development  and  performance.  A  critical 
purpose  of  DT&E  is  to  identify  the  risks  of 
development  by  testing  and  evaluating  se¬ 
lected  high-risk  components  or  subsystems. 
Development  test  and  evaluation  is  the 
developer's  tool  to  show  that  the  system 
performs  as  specified  or  that  deficiencies 
have  been  corrected  and  the  system  is  ready 
for  operational  testing  and  fielding  (Figure 
8-1).  The  DT&E  results  are  used  through¬ 
out  the  systems  engineering  process  to  pro¬ 
vide  valuable  data  in  support  of  formal 
design  reviews.  This  chapter  describes  the 
test's  relationship  to  the  formal  design  re¬ 
views  essential  to  the  systems  engineering 
process. 

8.2  DT&E  AND  THE  REVIEW 
PROCESS 

8.2.1  The  Technical  Review  Process 

Technical  reviews  and  audits  are  conducted 
by  the  government  and  the  contractor  as 
part  of  the  systems  engineering  process  to 
ensure  the  design  meets  the  system,  sub¬ 
system  and  software  specifications.  Each 
review  is  unique  in  its  timing  and  orienta¬ 
tion.  Some  reviews  build  on  previous  re¬ 
views  and  take  the  design  and  testing  effort 
one  step  closer  to  the  final  system  design  to 


satisfy  the  operational  concept/purpose  for 
the  weapon  system.  Table  8-1  illustrates 
the  sequencing  of  the  technical  reviews  in 
relation  to  the  test  and  evaluation  phases. 

The  review  process  was  established  to  en¬ 
sure  that  the  system  under  development 
would  meet  government  requirements.  The 
reviews  evaluate  data  from  contractor  and 
government  testing,  engineering  analysis, 
and  models  to  determine  if  the  system  or  its 
components  will  eventually  meet  all  func¬ 
tional  and  physical  specifications  and  to 
determine  the  final  system  design.  The  sys¬ 
tem  specification  is  very  important  in  this 
process.  It  is  the  document  used  as  a  bench 
mark  to  compare  contractor  progress  in 
designing  and  developing  the  desired  prod¬ 
uct.  Guidelines  for  these  formal  technical 
reviews  and  audits  can  be  found  in  EIA 
Standard  632  or  IEEE  1220-1994  (MIL-STD- 
1521B  cancelled). 

8.2.2  Testing  in  Support  of 
Technical  Reviews 

The  testing  community  must  be  continu¬ 
ally  involved  in  supporting  the  technical 
reviews  of  their  systems.  Decisions  made  at 
these  reviews  have  major  impacts  on  the 
system  test  design,  resources  required  to 
test,  and  the  development  of  the  Test  and 
Evaluation  Master  Plan  (TEMP)  and  other 
documentation.  A  more  detailed  discus¬ 
sion  of  testing  to  support  the  technical  re¬ 
views  is  provided  in  the  Systems  Engineer- 
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Figure  8-1 .  Relationship  of  DT&E  to  the  Acquisition  Process 


Table  8-1.  Technical  Reviews  and  Audits 


WHEN 

PURPOSE 

DOCUMENTATION/ 

DATA 

REQUIREMENTS 

REVIEW 

SRR 

LATECE 

-  EVALUATE  SYSTEM 
FUNCTIONAL  REQUIRE¬ 
MENTS 

-  PRELIM  PERF  SPEC 

-  PRELIM  PLANNING  DOCU¬ 
MENTATION 

-  FFBD,RAS,MBN  ANALYSIS 

SYSTEM 

FUNCTIONAL 

REVIEW 

SFR 

LATEPDRR 

-  EVALUATE  SYSTEM 

DESIGN 

-  VALIDATE  “A”  SPEC 

-  ESTABLISH  SYSTEM  LEVEL 
FUNCTIONAL  BASELINE 

-  PERFORMANCE  SPEC 

-  PRELIM  ITEM  (PERF)  SPEC 
>  DESIGN  DOCUMENTS 

-  RAS,SSD,TLS 

SOFTWARE 

SPECIFICATION 

REVIEW 

SSR 

EARLY  EMD 

PRIOR  SW  PDR 

-  EVALUTESW  PERFOR¬ 
MANCE  REQUIREMENTS 

-  VALIDATE  S/W  SPECS 

-  ESTABLISH  SW  SPECS 
BASELINE 

-  S/W  SPEC 

-  (SRS  &  IRS) 

-  OPS  CONCEPT  DOC 

PRELIMINARY 

DESIGN 

REVIEW 

PDR 

EARLY  EMD 

-  VALIDATE  ITEM  (PERF) 

SPECS 

-  ESTABLISH  HW  ALLO¬ 
CATED  BASELINE 

-  EVALUATE  PRELIMINARY 
DESIGN  HW&SW 

-ITEM  (PERF)  SPEC 

-  DES  DOC  TEST  PLAN 

-  ICD,ENGR  DRAWINGS 

-  PRELIMINARY  SDD-IDD 

CRITICAL 

DESIGN 

REVIEW 

CDR 

EARLY/MID 

EMD 

-  EVALUATE  Cl  DESIGN 

-  DETERMINE  READINESS 

FOR  FABRICATION 

-  PRELIM  ITEM  (DETAIL), 
PROCESS,  MATERIAL  SPECS 

-  DETAIL  DESIGN  DOCUMENTS 
INCLUDE  SDD-IDD 

TEST 

READINESS 

REVIEW 

TRR 

MID/LATE  EMD 

-  APPROVE  SW  TEST 
PROCEDURES 

-  DETERMINE  READINESS 

FOR  FORMAL  TEST 

-  SW  TEST  PLAN/ PROCE¬ 
DURES 

-  INFORMAL  SW  TEST 

RESULTS 

FUNCTIONAL 

CONFIGURATION 

AUDIT 

FCA 

LATE  EMD 

-  VERIFY  Cl  ACTUAL 

PERFORMANCE  COMPLIES 
WITH  HARDWARE  DEVEL¬ 
OPMENT  OR  SRS  &  IRS 

-  TEST  PUNS  &  DESCRIP¬ 
TIONS 

-  SOFTWARE  TEST  REPORTS 

FORMAL 

QUALIFICATION 

REVIEW 

FOR 

LATE  EMD 

-  VERIFY  Cl’S  PERFORM  IN 
SYSTEM  ENVIRONMENT 

-  TEST  REPORTS 

-  SPECS 

-  O&SDOCS 

PRODUCTION 

READINESS 

REVIEW 

PRR 

INCREMENTALLY 

EMD 

-  ASSESS  RISK  FOR 
PRODUCTION  GO-AHEAD 

-  PROD  PUNNING  DOCU¬ 
MENTS 

PHYSICAL 

CONRGURATION 

AUDIT 

PCA 

LATE  EMD 

EARLY  PROD 

-  FORMAT  EXAMINATION  OF 
THE  AS-BUILT 

-  FINAL  ITEM  (DETAIL)  SPEC 

-  LISTINGS 

-  LVL II  &  III  DRAWING 
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ing  Management  Guide  published  by  the 
Defense  Systems  Managment  College.  The 
reviews  focus  primarily  on  government 
technical  specifications  for  the  system.  Fig¬ 
ure  8-2  illustrates  the  program  specifica¬ 
tions  and  how  they  are  developed  in  the 
system  life  cycle. 

8.2.3  Formal  Reviews 

8.2.3.1  Systems  Requirements 
Review  (SRR) 

The  SRR  is  normally  conducted  during  the 
system  Concept  Exploration  and  Defini¬ 
tion  Phase  or  early  Program  Definition  and 
Risk  Reduction.  It  consists  of  a  review  of 
the  system/system  segment  specifications, 
previously  known  as  the  "A"  specifications 
(System  Functional  Block  Diagram,  Refer¬ 
ence  30,  Chapter  12),  and  is  conducted  after 
the  accomplishment  of  functional  analysis 
and  preliminary  requirements  allocation. 
EXiring  this  review,  the  systems  engineer¬ 
ing  management  activity  and  its  output  are 
reviewed  for  responsiveness  to  the  State¬ 
ment  of  Work  requirements.  The  primary 
function  of  the  SRR  is  to  ensure  that  system's 
requirements  have  been  completed  and 
properly  identified  and  that  there  is  a  mu¬ 
tual  understanding  between  the  contractor 
and  the  government.  During  the  review, 
the  contractor  describes  his  progress  and 
any  problems  in  risk  identification  and  rank¬ 
ing,  risk  avoidance  and  reduction,  trade¬ 
off  analysis,  producibility  and  manufac¬ 
turing  considerations,  and  hazards  consid¬ 
erations.  The  results  of  integrated  test  plan¬ 
ning  are  reviewed  to  ensure  the  adequacy 
of  planning  to  assess  the  design  and  to 
identify  risks.  Issues  of  testability  of  re¬ 
quirements  should  be  discussed. 

8.2.3.2  Systems  Functional  Review 
(SFR) 

The  SFR  is  conducted  as  a  final  review 
before  submittal  of  the  Phase  I  products  or 


as  the  initial  Engineering  and  Manufactur¬ 
ing  Phase  (EMD)  review.  The  system  speci¬ 
fication  is  validated  to  ensure  that  the  most 
current  specification  is  included  in  the  Sys¬ 
tem  Functional  Baseline  and  that  they  are 
adequate  and  cost-effective  to  satisfy  vali¬ 
dated  mission  requirements.  The  SFR  en¬ 
compasses  the  total  system  requirement  of 
operations,  maintenance,  test,  training, 
computers,  facilities,  personnel  and  logis¬ 
tics  considerations.  A  technical  understand¬ 
ing  should  be  reached  on  the  validity  and 
the  degree  of  completeness  of  specifica¬ 
tions,  design,  operational  concept  docu¬ 
mentation,  software  requirements  specifi¬ 
cations  and  interface  requirements  specifi¬ 
cations  during  this  review. 

8.2.3.3  Software  Specification 
Review  (SSR) 

The  SSR  is  a  formal  review  of  the  computer 
system  configuration  item  (CSCI)  require¬ 
ments,  normally  held  after  a  SFR  but  before 
the  start  of  a  CSCI  preliminary  design.  Its 
purpose  is  to  validate  the  allocated  baseline 
for  preliminary  CSCI  design  by  demon¬ 
strating  to  the  government  the  adequacy  of 
the  software  requirements  specifications, 
interface  requirements  specifications  and 
operational  concept  documentation. 

8.2.3.4  Preliminary  Design 
Review  (PDR) 

The  PDR  is  a  formal  technical  review  of  the 
basic  approach  for  a  configuration  item.  It 
is  conducted  at  the  configuration  item  and 
system  level  early  in  the  EMD  Phase  to 
confirm  that  the  preliminary  design  logi¬ 
cally  follows  SFR  findings  and  meets  the 
system  requirements.  The  review  results  in 
an  approval  to  begin  the  detailed  design. 
The  draft  item  specifications  (performance) 
are  reviewed  during  the  PDR.  The  purpose 
of  the  PDR  is  to:  evaluate  the  progress, 
technical  adequacy  and  risk  resolution  (on 
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Figure  8-2. 


technical,  cost  and  schedule  basis)  of  the 
configuration  item  (Cl)  design  approach; 
review  development  test  (DT)  and  opera¬ 
tional  test  (OT)  activities  to  measure  the 
performance  of  each  Cl;  and  establish  the 
existence  and  compatibility  of  the  physical 
and  functional  interface  among  the  Cl  and 
other  equipment. 

8.2.3.5  Critical  Design  Review  (CDR) 

The  CDR  may  be  conducted  on  eachh  Cl 
and/ or  at  the  system  level.  It  is  conducted 
during  the  EMD  Phase  when  the  detailed 
design  is  essentially  complete  and  prior  to 
the  FC  A.  During  the  CDR,  the  overall  tech¬ 
nical  program  risks  associated  with  each  Cl 
are  also  reviewed  on  a  technical,  cost  and 
schedule  basis.  It  includes  a  review  of  the 
item  specifications  (detail)  and  the  status  of 
both  the  system's  hardware  and  software. 
Input  from  qualification  testing  should  as¬ 
sist  in  determination  of  readiness  for  de¬ 
sign  freeze  and  low  rate  initial  production 
(LRIP). 

8.2.3.6  Test  Readiness  Review  (TRR) 

The  TRR  is  a  formal  review  of  the 
contractor's  readiness  to  begin  CSCI  test¬ 
ing.  A  government  witness  will  observe  the 
system  demonstration  to  verify  that  the 
system  is  ready  to  proceed  with  CSCI  test¬ 
ing.  It  is  conducted  after  the  software  test 
procedures  are  available  and  computer 
software  components  testing  is  complete. 
The  purpose  of  the  TRR  is  for  the  program 
management  office  (PMO)  to  determine 
whether  the  contractor  is  ready  to  begin 
CSCI  testing. 

8.2.3.7  Functional  Configuration 
Audit  (FCA) 

The  FCA  is  a  formal  review  to  verify  that 
the  Cl's  performance  complied  with  its  sys¬ 
tem  specification.  The  item  specifications 


are  derived  from  the  system  requirements 
and  baseline  documentation.  During  the 
FCA,  all  relevant  test  data  is  reviewed  to 
verify  that  the  item  has  performed  as  re¬ 
quired  by  its  functional  and/or  allocated 
configuration  identification.  The  audit  is 
conducted  on  the  item  representative  (pro¬ 
totype  or  production)  of  the  configuration 
to  be  released  for  production.  The  audit 
consists  of  a  review  of  the  contractor's  test 
procedures  and  results.  Informationprovided 
will  be  used  by  the  FCA  to  determine  the 
status  of  planned  tests. 

8.2.3.8  Physical  Configuration 
Audit  (PCA) 

The  PCA  is  a  formal  review  which  estab¬ 
lishes  the  product  baseline  as  reflected  in 
an  early  production  Cl.  It  is  the  examina¬ 
tion  of  the  as-built  version  of  hardware  and 
software  CIs  against  its  technical  docu¬ 
mentation.  The  PCA  also  determines  that 
the  acceptance  testing  requirements  pre¬ 
scribed  by  the  documentation  are  adequate 
for  acceptance  of  production  units  of  a  Cl 
by  quality  assurance  activities.  It  includes  a 
detailed  audit  of  engineering  drawings, 
final  Part  II  item  specifications  (detail),  tech¬ 
nical  data  and  plans  for  testing  that  will  be 
utilized  during  production.  The  PCA  is 
performed  on  all  first  articles  and  on  the 
first  CIs  delivered  by  a  new  contractor. 

8.2.3.9  Formal  Qualification 
Review  (FQR) 

The  FQR  is  a  systems-level  configuration 
audit  that  may  be  conducted  after  system 
testing  is  completed.  The  objective  is  to 
verify  that  the  actual  performance  of  the  Cl, 
as  determined  through  testing,  complies 
with  its  item  specifications  (performance) 
and  to  document  the  results  of  the  qualifi¬ 
cation  tests.  The  FQR  and  FCA  are  often 
performed  at  the  same  time;  however,  if 
sufficient  test  results  are  not  available  at  the 
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FCA  to  ensure  the  Cl  will  perform  in  its 
operational  environment,  the  FQR  can  be 
scheduled  for  a  later  time. 

8.2.3.10  Production  Readiness 
Review  (PRR) 

The  PRR  is  an  assessment  of  the  contractor's 
ability  to  produce  the  items  on  the  contract. 
It  is  usually  a  series  of  reviews  conducted 
before  an  LRIP  or  full-scale  production 
decision.  For  more  information,  see  Chap¬ 
ter  10,  Production  Related  Testing  Activi¬ 
ties. 

8.2.3.11  Configuration  Change  Control 

The  Configuration  Change  Control  review 
is  an  assessment  of  the  impact  of  engineer¬ 
ing  or  design  changes.  It  is  conducted  by 
the  engineering,  test  and  evaluation  (T&E) 
and  program  manager  (PM)  portions  of  the 
PMO.  Most  approved  Class  I  engineering 
change  proposals  will  require  additional 
testing,  and  the  test  manager  must  accom¬ 
modate  the  new  schedules  and  resource 
requirements.  Adequate  testing  must  be 
accomplished  to  ensure  integration  and 
compatibility  of  these  changes.  For  example, 
an  engineering  change  review  was  con¬ 
ducted  to  replace  the  black  and  white 


monitors  and  integrate  color  monitors  into 
the  Airborne  Warning  and  Control  System 
(AW  ACS).  Further,  the  AWACS  operating 
software  had  to  be  upgraded  to  handle 
color  enhancement.  The  review  was  con¬ 
ducted  by  the  government  PMO;  and  sec¬ 
tions  of  the  PMO  were  tasked  to  contract, 
test,  engineer,  logistically  support,  control, 
cost  and  finance  the  change  to  completion. 
Guidelines  for  configuration  control  and 
engineering  changes  are  discussed  in  El  A/ 
IS-649  (MIL-STD-480  cancelled). 

8.3  SUMMARY 

Design  reviews  are  an  integral  and  essen¬ 
tial  part  of  the  systems  engineering  pro¬ 
cess.  The  meetings  range  from  very  formal 
reviews  by  government  and  contractor  PMs 
to  informal  technical  reviews  concerned 
with  product  or  task  elements  of  the  work 
breakdown  structure.  Reviews  may  be  con¬ 
ducted  in  increments  over  time.  All  re¬ 
views  share  the  common  objective  of  deter¬ 
mining  the  technical  adequacy  of  the  exist¬ 
ing  design  to  meet  technical  requirements. 
The  DT/OT  assessments  and  test  results 
are  made  available  to  the  reviews,  and  it  is 
important  that  the  test  community  be  in¬ 
volved. 
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COMBINED  AND  CONCURRENT 
TESTING 


9.1  INTRODUCTION 

The  terms  "concurrency,"  "concurrent  test¬ 
ing,"  and  "combined  testing"  are  sometimes 
subject  to  misinterpretation.  Concurrency 
is  defined  as  an  approach  to  system  devel¬ 
opment  and  acquisition  in  which  phases  of 
the  acquisition  process,  which  normally 
occur  sequentially,  overlap  to  some  extent. 
For  example,  a  weapon  system  enters  the 
production  phase  while  development  ef¬ 
forts  are  still  underway. 

Concurrent  testing  refers  to  circumstances 
when  development  testing  and  operational 
testing  take  place  at  the  same  time  as  two 
parallel  but  separate  and  distinct  activities. 
In  contrast,  combined  testing  refers  to  a 
single  test  program  conducted  to  support 
development  test  (DT)  and  operational  test 
(OT)  objectives.  This  chapter  discusses  the 
use  of  combined  testing  and  concurrent 
testing,  and  highlights  some  of  the  advan¬ 
tages  and  disadvantages  associated  with 
these  approaches.  (Table  9-1) 

9.2  COMBINING  DEVELOPMENT 
TEST  AND  OPERATIONAL  TEST 

Certain  test  events  can  be  organized  to 
provide  information  useful  to  develop¬ 
ment  testers  and  operational  testers.  For 
example,  a  prototype  free-fall  munition 
could  be  released  from  a  fighter  aircraft 
at  operational  employment  conditions 
instead  of  from  a  static  stand  to  satisfy  DT 
and  OT  objectives.  Such  instances  need  to 


be  identified  to  prevent  unnecessary  du¬ 
plication  of  effort  and  to  control  costs.  A 
combined  testing  approach  is  also  appro¬ 
priate  for  certain  specialized  types  of  test¬ 
ing.  For  example,  in  the  case  of  nuclear 
survivability  and  hardness  testing,  sys¬ 
tems  cannot  be  tested  in  a  totally  realistic 
operational  environment;  therefore,  a 
single  test  program  is  often  used  to  meet 
both  development  and  operational  test 
objectives. 

The  Department  of  Defense  (DoD)  5000.2- 
R  encourages  combined  testing  suggesting 
that  a  combined  development  test  and 
evaluation  (DT&E)  and  operational  test  and 
evaluation  (OT&E)  approach  should  be  con¬ 
sidered  when  there  are  time  and  cost  sav¬ 
ings.  The  combined  approach  must  not 
compromise  either  DT  or  OT  objectives.  If 
this  approach  is  elected,  planning  efforts 
must  be  carefully  coordinated  early  in  the 
program  to  ensure  data  is  obtained  to  sat¬ 
isfy  the  needs  of  both  the  developing  agency 
and  the  independent  operational  tester. 
Care  must  also  be  exercised  to  ensure  a 
combined  test  program  contains  dedicated 
OT  events  to  satisfy  the  requirement  for  an 
independent  evaluation.  A  final  indepen¬ 
dent  phase  of  OT&E  testing  shall  be  re¬ 
quired  for  beyond  low  rate  initial  produc¬ 
tion  (BLRIP)  decisions.  In  all  combined  test 
programs,  provisions  for  separate  indepen¬ 
dent  development  and  operational  evalua¬ 
tions  of  test  results  should  be  provided. 
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Table  9-1 .  Combined  vs.  Concurrent  Testing:  Advantages  and  Limitations 
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Service  regulations  describe  the  sequence 
of  activities  in  a  combined  testing  program 
as  follows; 

Although  OT&E  is  separate  and  dis¬ 
tinct  from  DT&E,  most  of  the  gener¬ 
ated  data  are  mutually  beneficial  and 
freely  shared.  Similarly,  the  resources 
needed  to  conduct  and  support  both 
test  efforts  are  often  the  same  or  very 
similar.  Thus,  when  sequential  DT&E 
and  OT&E  efforts  would  cause  delay 
or  increase  the  acquisition  cost  of  the 
system,  DT&E  and  OT&E  are  com¬ 
bined.  When  combined  testing  is 
planned,  the  necessary  test  condi¬ 
tions  and  data  required  by  both  DT &E 
and  OT&E  organizations  must  be  in¬ 
tegrated.  Combined  testing  can  nor¬ 
mally  be  divided  into  three  segments. 

In  the  first  segment,  DT&E  event[s] 
usually  assume  priority  because  criti¬ 
cal  technical  and  engineering  tests  must 
be  accomplished  to  continue  the  engi¬ 
neering  and  development  process. 
During  this  early  period,  OT&E  per- 
sonnel  participate  to  gain  familiarity 
with  the  system  and  to  gain  access  to 
any  test  data  that  can  support  OT&E. 
Next,  the  combined  portion  of  the  test¬ 
ing  frequently  includes  shared  objec¬ 
tives  or  joint  data  requirements.  The 
last  segment  normally  contains  the 
dedicated  OT&E  or  separate  OT&E 
events  to  be  conducted  by  the  OT&E 
agency.  The  OT&E  agency  and  imple¬ 
menting  command  must  ensure  the 
combined  test  is  planned  and  executed 
to  provide  the  necessary  operational 
test  information.  The  OT&E  agency 
provides  an  independent  evaluation 
of  the  OT&E  portion  and  is  ultimately 
responsible  for  achieving  OT &E  objec¬ 
tives. 

The  testing  of  the  Navy’s  F-14  aircraft  has 
been  cited  as  an  example  of  a  successful 


combined  test  and  evaluation  (T&E)  pro¬ 
gram  (Reference  110).  A  key  factor  in  the 
success  of  the  F-14  approach  was  the  selec¬ 
tion  of  a  T&E  coordinator  responsible  for 
supervising  the  generation  of  test  plans 
that  integrated  the  technical  requirements 
of  the  developers  with  the  operational  re¬ 
quirements  of  the  users.  The  T&E  coordi¬ 
nator  was  also  responsible  for  the  alloca¬ 
tion  of  test  resources  and  the  overall  man¬ 
agement  of  the  test.  In  a  paper  for  the 
Defense  Systems  Management  College,  Mr. 
Thomas  Hoivik  describes  the  successful 
F-14  test  program  as  follows; 

"The  majority  of  the  Navy  develop¬ 
mental  and  operational  testing  took 
place  during  the  same  period  and  even 
on  the  same  flights.  Maximum  use  was 
made  of  contractor  demonstrations 
witnessed  by  the  Navy  testing  activi¬ 
ties  to  obviate  the  retesting  of  a  techni¬ 
cal  point  already  demonstrated  by  the 
contractor.  Witnessing  by  testing  ac¬ 
tivities  was  crucially  important  and 
allowed  the  contractor's  data  to  be 
readily  accepted  by  the  testing  activi¬ 
ties.  Tins  approach  also  helped  to  elimi¬ 
nate  redundancy  in  testing,  i.e.  the 
testing  of  the  same  performance  pa¬ 
rameter  by  several  different  activities 
which  has  been  a  consistent  and  waste¬ 
ful  feature  of  Navy  testing  in  the  past." 

Obviously,  this  approach  placed  a  great 
deal  of  responsibility  directly  on  the  shoul¬ 
ders  of  the  T&E  Coordinator,  and  required 
the  T&E  Coordinator's  staff  to  deal  knowl¬ 
edgeably  with  a  wide-ranging  and  com¬ 
plex  test  plan. 

9.3  CONCURRENT  TESTING 

In  1983,  a  senior  DoD  T&E  official  testified 
that  a  concurrent  testing  approach  is  usu¬ 
ally  not  an  effective  strategy  (Reference  105). 
He  acknowledged,  however,  that  certain  test 
events  may  provide  information  useful  to 
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development  and  operational  testers,  and 
test  planners  must  be  alert  to  identify  those 
events.  His  testimony  included  the  follow¬ 
ing  examples  of  situations  where  a  concur¬ 
rent  testing  approach  was  unsuccessful: 

(1)  During  A  AH  (Advanced  Attack 
Helicopter)  testing  in  1981,  the  Target 
Acquisition  Designation  System 
(TADS)  was  undergoing  developmen¬ 
tal  and  operational  testing  at  the  same 
time.  The  schedule  did  not  allow 
enough  time  for  qualification  testing 
(a  development  test  activity)  of  the 
TADS  prototype  prior  to  a  full  field 
test  of  the  total  aircraft  system,  nor  was 
there  time  to  introduce  changes  to 
TADS  problems  discovered  in  tests. 
As  a  result,  the  TADS  performed  poorly 
and  was  unreliable  during  the  opera¬ 
tional  test.  The  resulting  DSARC  [De¬ 
fense  Systems  Acquisition  Review 
Council]  action  required  the  Army  to 
fix  and  retest  the  TADS  prior  to  release 
of  second  year  and  subsequent  pro¬ 
duction  funds. 

(2)  When  the  AIM-7  Sparrow  air-to- 
air  missile  was  tested,  an  attempt  was 
made  to  move  into  operational  testing 
while  developmental  reliability  test¬ 
ing  was  still  underway.  The  opera¬ 
tional  test  was  suspended  after  less 
than  two  weeks  because  of  poor  reli¬ 
ability  of  the  test  missiles.  The  pro¬ 
gram  concentrated  on  an  intensive 
reliability  improvement  effort.  A  year 
after  the  initial  false  start,  a  full  opera¬ 


tional  test  was  conducted  and  com¬ 
pleted  successfully. 

(3)  The  Maverick  missile  had  a  similar 
experience  of  being  tested  in  an  opera¬ 
tional  environment  before  component 
reliability  testing  was  completed.  As  a 
result,  reliability  failures  had  a  major 
impact  on  the  operational  testers  and 
resulted  in  the  program  being  ex¬ 
tended. 

9.4  ADVANTAGES 
AND  LIMITATIONS 

Before  adopting  a  combined  or  concurrent 
testing  approach,  program  and  test  manag¬ 
ers  are  advised  to  consider  the  advantages 
and  disadvantages  summarized  in  Table  9-1 . 

9.5  SUMMARY 

A  combined  or  concurrent  testing  approach 
may  offer  an  effective  means  of  shortening 
the  time  required  for  testing  and  achieving 
cost  savings.  If  such  an  approach  is  used, 
extensive  coordination  is  required  to  en¬ 
sure  the  development  and  operational  re¬ 
quirements  are  addressed. 

It  is  possible  to  have  combined  test  teams, 
consisting  of  DT&E,  OT&E  and  contractor 
personnel,  involved  throughout  the  testing 
process.  The  teams  can  provide  mutual 
support  and  share  mutually  beneficial  data 
as  long  as  the  test  program  is  carefully 
planned  and  executed  and  reporting  ac¬ 
tivities  are  conducted  separately. 
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PRODUCTION-RELATED 
TESTING  ACTIVITIES 


10.1  INTRODUCTION 

Most  of  the  test  and  evaluation  (T&E)  dis¬ 
cussed  in  this  guidebook  concerns  the  test¬ 
ing  of  the  actual  weapon  or  system  being 
developed,  but  the  program  manager  (PM) 
must  also  evaluate  production-related  test 
activities  and  the  production  process.  This 
chapter  describes  production  management 
and  the  production  process  testing  required 
to  ensure  the  effectiveness  of  the  manufac¬ 
turing  process  and  the  producibility  of  the 
system's  design. 

Normally,  the  development  test  (DT)  and 
operational  test  (OT)  organizations  are  not 
involved  directly  in  this  process.  Usually, 
the  manufacturing  and  quality  assurance 
sections  of  the  program  office  and  a  repre¬ 
sentative  of  the  government  Defense  Con¬ 
tract  Management  Command  (DCMC) 
oversee/perform  many  of  these  functions. 

10.2  PRODUCTION  MANAGEMENT 

Production  (manufacturing)  management 
is  defined  as  "the  effective  use  of  resources 
to  produce,  on  schedule,  the  required  num¬ 
ber  of  end  items  that  meet  specified  quality, 
performance,  and  cost.  Production  man¬ 
agement  includes,  but  is  not  limited  to, 
industrial  resource  analysis,  producibility 
assessment,  producibility  engineering  and 
planning,  production  engineering,  indus¬ 
trial  preparedness  planning,  post-produc¬ 
tion  planning,  and  productivity  enhance¬ 
ment,"  (Department  of  Defense  (DoD)  Di¬ 
rective  5000.34).  Production  management 


begins  early  in  the  acquisition  process,  as 
early  as  the  Concept  Exploration  Phase, 
and  is  specifically  addressed  at  each  pro¬ 
gram  milestone  (MS)  decision  point.  For 
instance,  during  Phase  0,  production  feasi¬ 
bility,  costs  and  risks  should  be  addressed. 
Before  MS  I,  the  PM  must  conduct  an  in¬ 
dustrial  resource  analysis  (IRA)  to  deter¬ 
mine  the  availability  of  production  re¬ 
sources  (e.g.,  capital,  material,  manpower) 
required  to  support  the  production  of  the 
weapon  system.  On  the  basis  of  the  results 
of  the  IRA,  critical  materials,  deficiencies  in 
the  U.S.  industrial  base  and  requirements 
for  new  or  updated  manufacturing  tech¬ 
nology  can  be  identified.  Analysis  of  the 
industrial-base  capacity  is  one  of  the  con¬ 
siderations  in  preparing  for  the  MS  I  deci¬ 
sion.  As  development  proceeds,  the  manu¬ 
facturing  strategy  is  developed;  and  de¬ 
tailed  plans  are  made  for  Production,  Phase 
in.  Independent  producibility  assessments, 
conducted  in  preparation  for  the  transition 
from  development  to  production,  are  re¬ 
viewed  at  the  MS  II  decision  point.  At  MS  II, 
the  Engineering  and  Manufacturing  De¬ 
velopment  (EMD)  decision,  the 
producibility  of  the  system  design  concept 
is  evaluated  to  verify  that  the  system  can  be 
manufactured  in  compliance  with  the  pro¬ 
duction-cost  and  the  industrial-base  goals 
and  thresholds. 

The  MS  III  decision  is  supported  by  an 
assessment  of  the  readiness  of  the  system  to 
enter  production.  The  system  cannot  enter 


Phase  III  until  it  is  determined  the  princi¬ 
pal  contractors  have  the  necessary  re¬ 
sources  (i.e.,  physical,  financial,  and 
managerial  capacity)  to  achieve  the  cost 
and  schedule  commitments  and  to  meet 
peacetime  and  mobilization  requirements 
for  production  of  the  system.  The  method 
of  assessing  production  readiness  in 
preparation  for  MS  III  is  the  Production 
Readiness  Review  (PRR),  which  is  con¬ 
ducted  by  the  PM  and  staff. 

10.3  PRODUCTION  READINESS 
REVIEW  (PRR) 

The  following  are  guidelines  for  PRRs: 

This  review  is  intended  to  determine 
the  status  of  completion  of  the  specific 
actions  which  must  be  satisfactorily 
accomplished  prior  to  executing  a  pro¬ 
duction  go-ahead  decision.  The  review 
is  accomplished  in  an  incremental  fash¬ 
ion  during  the  Engineering  and  Manu¬ 
facturing  Phase,  usually  two  initial 
reviews  and  one  final  review  to  assess 
the  risk  in  exercising  the  production 
go-ahead  decision.  In  its  earlier  stages 
the  PRR  concerns  itself  with  gross  level 
manufacturing  concerns  such  as  the 
need  for  identifying  high  risk/low 
yield  manufacturing  processes  or  ma¬ 
terials  or  the  requirement  for  manu¬ 
facturing  development  effort  to  sat¬ 
isfy  design  requirements.  Timing  of 
the  incremental  PRR's  is  a  function  of 
program  posture  and  is  not  specifi¬ 
cally  locked  into  other  reviews. 

The  conduct  of  a  PRR  (Table  10-1)  is  the 
responsibility  of  the  PM,  who  usually  ap¬ 
points  a  director.  The  director  assembles  a 
team  comprised  of  individuals  in  the  disci¬ 
plines  of  design,  industry,  manufacturing, 
procurement,  inventory  control,  contracts, 
engineering  and  quality  training.  The  PRR 
director  organizes  and  manages  the  team 
effort  and  supervises  preparation  of  the 


findings.  The  PRR  is  conducted  as  a  time- 
phased  effort  during  the  EMD  Phase  fol¬ 
lowing  the  guidelines  presented  in  DoD 
5000.2-R. 

10.4  QUALIFICATION  TESTING 

Qualification  testing  is  performed  to  verify 
the  design  and  manufacturing  process,  and 
it  provides  a  baseUne  for  subsequent  accep¬ 
tance  tests.  The  production  qualification 
testing  is  conducted  at  the  unit,  subsystem 
and  system  level  on  production  items  and 
is  completed  before  the  production  deci¬ 
sion.  The  results  of  these  tests  are  a  critical 
factor  in  assessing  the  system's  readiness 
for  production.  Down  line  production  quali¬ 
fication  tests  are  performed  to  verify  pro¬ 
cess  control  and  may  be  performed  on  se¬ 
lected  parameters  rather  than  at  the  levels 
originally  selected  for  qualification. 

10.4.1  Production  Qualification 
Tests  (PQT) 

Production  qualification  tests  are  a  series  of 
formal  contractual  tests  conducted  to  en¬ 
sure  design  integrity  over  the  specified 
operational  and  environmental  range.  The 
tests  are  conducted  on  preproduction  items 
fabricated  to  the  proposed  production  de¬ 
sign  drawings  and  specifications.  The  PQTs 
include  all  contractual  reliability  and  main¬ 
tainability  demonstration  tests  required 
prior  to  production  release.  For  volume 
acquisitions,  these  tests  are  a  constraint  to 
production  release. 

10.4.2  First  Article  Tests  (FAT) 

First  article  tests  consist  of  a  series  of  formal 
contractual  tests  conducted  to  ensure  the 
effectiveness  of  the  manufacturing  process, 
equipment  and  procedures.  These  tests  are 
conducted  on  a  random  sample  from  the 
first  production  lot.  These  series  of  tests 
are  repeated  if  the  manufacturing  pro¬ 
cess,  equipment  or  procedure  is  changed 
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significantly  and  when  a  second  or  alterna¬ 
tive  source  of  manufacturing  is  brought  on 
line,  [Federal  Acquisition  Regulation  (FAR) 
Part  9.3] 

10.5  TRANSITION  TO  PRODUCTION 

In  an  acquisition  process,  often  the  first 
indication  that  a  system  will  experience 
problems  is  during  the  transition  from  EMD 
to  low  rate  initial  production  (LRIP).  This 
transition  continues  over  an  extended  pe¬ 
riod,  often  months  or  years;  and  during  this 
period,  the  system  is  undergoing  stringent 
contractor  and  government  testing.  There 
may  be  unexpected  failures  requiring  sig¬ 
nificant  design  changes,  which  impact  on 
quality,  producibility,  supportability  and 
may  require  program  schedule  slippage. 

Long  periods  of  transition  usually  indicate 
that  insufficient  attention  to  design  or 
producibility  was  given  early  in  the  Con¬ 
cept  Exploration  (CE)  or  Program  Defini¬ 
tion  and  Risk  Reduction  phases. 

10.5.1  Transition  Planning 

Producibility  engineering  and  planning 
(PEP)  is  the  common  thread  that  guides  a 
system  from  CE  to  production.  Planning  is 
a  management  tool  used  to  ensure  that 
adequate  risk-handling  measures  have  been 
taken  to  transition  from  development  to 
production.  It  contains  a  checklist  to  be 
used  during  the  readiness  reviews.  Plan¬ 
ning  should  tie  together  the  applications  of 
designing,  testing  and  manufacturing  ac¬ 
tivities  to  reduce  data  requirements,  dupli¬ 
cation  of  effort,  costs  and  scheduling  and  to 
ensure  early  success  of  the  EMD  first  pro¬ 
duction  article. 

10.5.2  Testing  During  the  Transition 

Testing  accomplished  during  the  transi¬ 
tion  from  development  to  production  will 
include  acceptance  testing,  manufacturing 


screening  and  final  testing.  These  technical 
tests  are  performed  by  the  contractor  to  en¬ 
sure  the  system  will  transition  smoothly  and 
that  test  design  and  manufacturing  issues 
affecting  design  are  addressed.  During  this 
same  period,  the  government  will  be  using 
the  latest  available  configuration  item  to  con¬ 
duct  the  initial  operational  test  and  evalua¬ 
tion  (lOT&E).  The  impact  of  these  tests  may 
overwhelm  the  configuration  management 
of  the  system  unless  careful  planning  is  ac¬ 
complished  to  handle  these  changes. 

10.6  LOW  RATE  INITIAL 
PRODUCTION  (LRIP) 

Low  rate  initial  production  is  the  produc¬ 
tion  of  a  system  in  limited  quantity  to  pro¬ 
vide  articles  for  lOT&E  and  to  demonstrate 
production  capability.  Also,  it  permits  an 
orderly  increase  in  the  production  rate  suf¬ 
ficient  to  lead  to  full-rate  production  upon 
successful  completion  of  operational  test¬ 
ing.  The  decision  to  have  an  LRIP  is  made 
at  the  Milestone  (MS)  II  approval  of  the 
program  acquisition  strategy.  At  that  time, 
the  PM  must  identify:  (1)  the  quantity  to  be 
produced  during  LMP  and  (2)  the  quantity 
of  LRIP  articles  to  be  used  for  lOT&E  (Ac¬ 
quisition  Category  (ACAT)  I  approved  by 
the  Director,  Operational  Test  and  Evalua¬ 
tion  (DOT&E),  ACAT  11  and  HI  approved 
by  Service  Operational  Test  Agency  (OTA)) . 
When  the  decision  authority  thinly  the  sys¬ 
tems  will  not  perform  to  expectation,  the 
PM  may  direct  that  it  not  proceed  into  LRIP 
until  there  is  a  program  review.  The  DOT&E 
submits  a  report,  on  all  oversight  systems, 
to  congressional  committees  before  the  MS 
III  decision  approving  the  system  to  pro¬ 
ceed  beyond  LRIP  is  made. 

10.7  PRODUCTTION  ACCEPTANCE 
TEST  AND  EVALUATION  (PAT&E) 

Production  acceptance  test  and  evaluation 
ensures  that  production  items  demonstrate 
the  fulfillment  of  the  requirements  and 
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specifications  of  the  procuring  contract  or 
agreements.  The  testing  also  ensures  the 
system  being  produced  demonstrates  the 
same  performance  as  the  preproduction 
models.  The  procured  items  or  system  must 
operate  in  accordance  with  system  and  item 
specifications.  The  PAT&E  is  usually  con¬ 
ducted  by  the  program  office  quality  assur¬ 
ance  section  at  the  contractor's  plant  and 
may  involve  operational  users. 

For  example,  for  the  Rockwell  B-IB  Bomber 
production  acceptance,  Rockwell  and  Air 
Force  quality  assurance  inspectors  reviewed 
all  manufacturing  and  ground  testing  re¬ 
sults  for  each  aircraft.  In  addition,  a  flight 
test  team,  composed  of  contractor  and  Air 
Force  test  pilots,  flew  each  aircraft  a  mini¬ 
mum  of  10  hours,  demonstrating  all  on¬ 
board  aircraft  systems  while  in  flight.  Any 
discrepancies  in  flight  were  noted,  corrected 
and  tested  on  the  ground;  they  were  then 
retested  on  subsequent  checkouts  and  ac¬ 


ceptance  flights.  Once  each  aircraft  had 
passed  all  tests  and  all  systems  were  fully 
operational.  Air  Force  authorities  accepted 
the  aircraft.  The  test  documentation  also 
became  part  of  the  delivered  package.  Dur¬ 
ing  this  test  period,  the  program  office 
monitored  each  aircraft's  daily  progress. 

10.8  SUMMARY 

A  primary  purpose  of  production-related 
testing  is  to  lower  the  production  risk  in  a 
major  defense  acquisition  program.  The 
PM  must  ensure  the  contractor's  manufac¬ 
turing  strategy  and  capabilities  will  result 
in  the  desired  product  within  acceptable 
cost.  The  LRIP  and  PAT&E  also  play  major 
roles  in  ensuring  the  production  unit  is 
identical  to  the  design  drawings,  conforms 
to  the  specifications  of  the  contract,  and 
lOT&E  is  conducted  with  representative 
system  configurations. 
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Table  10-1.  PRR  Guidelines  Checklist 


PRODUCT  DESIGN 

•  PRODUCIBLE  AT  LOW  RISK 

•  STABILIZED  AT  LOW  RATE  OF  CHANGE 

•  VALIDATED 

•  RELIABILITY,  MAINTAINABILITY  AND  PERFORMANCE  DEMONSTRATED 

•  COMPONENTS  ENGINEERING  HAS  APPROVED  ALL  PARTS  SELECTIONS 

INDUSTRIAL  RESOURCES 

•  ADEQUATE  PLANT  CAPACITY  (PEACETIME  AND  WARTIME  DEMANDS) 

•  FACILITIES,  SPECIAL  PRODUCTION  AND  TEST  EQUIPMENT,  AND  TOOLING  IDENTIFIED 

•  NEEDED  PLANT  MODERNIZATION  (CAD/CAM,  OTHER  AUTOMATION)  ACCOMPLISHED,  WHICH  PRODUCES  AN 
INVESTED  CAPTIVE  PAYBACK  IN  TWO  TO  FIVE  YEARS 

•  ASSOCIATED  COMPUTER  SOFTWARE  DEVELOPED 

•  SKILLED  PERSONNEL  AND  TRAINING  PROGRAMS  AVAILABLE 

PRODUCTION  ENGINEERING  AND  PUNNING 

•  PRODUCTION  PLAN  DEVELOPED  (REFERENCE  MIL-STD-1528) 

•  PRODUCTION  SCHEDULES  COMPATIBLE  WITH  DELIVERY  REQUIREMENTS 

•  MANUFACTURING  METHODS  AND  PROCESSES  INTEGRATED  WITH  FACILITIES,  EQUIPMENT,  TOOLING  AND 
PLANT  LAYOUT 

•  VALUE  ENGINEERING  APPLIED 

•  ALTERNATE  PRODUCTION  APPROACHES  AVAILABLE 

•  DRAWINGS,  STANDARDS  AND  SHOP  INSTRUCTIONS  ARE  EXPLICIT 

•  CONFIGURATION  MANAGEMENT  ADEQUATE 

•  PRODUCTION  POLICIES  AND  PROCEDURES  DOCUMENTED 

•  MANAGEMENT  INFORMATION  SYSTEM  ADEQUATE 

•  CONTRACTOR’S  MANAGEMENT  STRUCTURE  IS  ACCEPTABLE  TO  THE  PMO 

•  THE  PEP  CHECKLIST  HAS  BEEN  REVIEWED 

MATERIALS 

•  ALL  SELECTED  MATERIALS  APPROVED  BY  CONTRACTOR’S  MATERIEL  ENGINEERS 

•  BILL  OF  MATERIALS  PREPARED 

•  “MAKE-OR-BUY”  DECISIONS  COMPLETE 

•  PROCUREMENT  OF  LONG  LEAD-TIME  ITEMS  IDENTIFIED 

•  SOLE-SOURCE  AND  GOVERNMENT-FURNISHED  ITEMS  IDENTIFIED 

•  CONTRACTOR’S  INVENTORY-CONTROL  SYSTEM  ADEQUATE 

•  CONTRACTOR’S  MATERIAL  COST  PROCUREMENT  PUN  COMPLETE 

QUALITY  ASSURANCE  (QA) 

•  QUALITY  PUN  IN  ACCORDANCE  WITH  CONTRACT  REQUIREMENTS 

•  QUALITY  CONTROL  PROCEDURES  AND  ACCEPTANCE  CRITERIA  ESTABLISHED 

•  QA  ORGANIZATION  PARTICIPATES  IN  PRODUCTION  PUNNING  EFFORT 

LOGISTICS 

•  OPERATIONAL  SUPPORT,  TEST  AND  DIAGNOSTIC  EQUIPMENT  AVAIUBLE  AT  SYSTEM  DEPLOYMENT 

•  TRAINING  AIDS,  SIMUUTORS  AND  OTHER  DEVICES  READY  AT  SYSTEM  DEPLOYMENT 

•  SPARES  INTEGRATED  INTO  PRODUCTION  LOT  FLOW 
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MODULE 

Operational 
Test  and  Evaluation 


Operational  Test  and  Evaluation  (OT&E)  is 
conducted  to  ensure  a  weapon  system  meets 
the  validated  requirements  of  the  user  in  a 
realistic  scenario.  Operational  tests  are  fo¬ 
cused  on  operational  requirements,  effec¬ 
tiveness  and  suitability,  and  not  on  the 
proof  of  engineering  specifications,  as  is 
the  case  with  development  testing.  This 
module  provides  an  overview  of  OT&E 
and  discusses  how  OT&E  results  provide 
essential  information  for  milestone  deci¬ 
sions. 


INTRODUCTION  TO  OPERATIONAL 
TEST  AND  EVALUATION 


11.1  INTRODUCTION 

This  chapter  provides  an  introduction  to 
the  concept  of  operational  test  and  evalua¬ 
tion  (OT&E).  It  outlines  the  purpose  of 
OT&E,  discusses  the  primary  participants 
in  the  OT&E  process,  describes  several  types 
of  OT&E,  and  includes  some  general  guide¬ 
lines  for  the  successful  planning,  execution 
and  reporting  of  OT&E  programs. 

11.2  PURPOSE  OF  OT&E 

Operational  test  and  evaluation  is  con¬ 
ducted  for  major  programs  by  an  organiza¬ 
tion  that  is  independent  of  the  developing, 
procuring  and  using  commands.  Some  form 
of  operational  assessment  is  normally  con¬ 
ducted  in  each  acquisition  phase.  Each  as¬ 
sessment  should  be  keyed  to  a  decision 
review  in  the  materiel  acquisition  process. 
It  should  include  typical  user  operators, 
crews  or  units  in  realistic  combat  simula¬ 
tions  of  operational  environments.  The 
OT&E  provides  the  decision  authority  with 
an  estimate  of: 

(1)  The  degree  of  satisfaction  of  the  user's 
requirements  expressed  as  operational  ef¬ 
fectiveness  and  operational  suitability  of 
the  new  system; 

(2)  The  system's  desirability,  consider¬ 
ing  equipment  already  available,  and  op¬ 
erational  benefits  or  burdens  associated 
with  the  new  system; 

(3)  The  need  for  further  development  of 
the  new  system  to  correct  performance  de¬ 
ficiencies; 


(4)  The  adequacy  of  doctrine,  organiza¬ 
tions,  operating  techniques,  tactics  and 
training  for  employment  of  the  system;  of 
maintenance  support  for  the  system;  and  of 
the  system's  performance  in  the  counter¬ 
measures  environment. 

11.3  TEST  PARTICIPANTS 

The  OT&E  of  developing  systems  is  man¬ 
aged  by  an  independent  operational  test¬ 
ing  agency,  which  each  Service  is  required 
to  maintain.  It  is  accomplished  under  con¬ 
ditions  of  operational  realism  whenever 
possible.  Personnel  who  operate,  maintain 
and  support  the  system  during  OT&E  are 
trained  to  a  level  commensurate  with  that 
of  personnel  who  will  perform  these  func¬ 
tions  under  peacetime  and  wartime  condi¬ 
tions.  Also,  program  management  office 
(PMO)  personnel,  the  integrated  product 
teams,  and  test  coordinating  groups  play 
important  parts  in  the  overall  OT&E  plan¬ 
ning  and  execution  process. 

11.3.1  Service  Operational  Test 
Agencies 

The  operational  test  and  evaluation  agen¬ 
cies  (OTE A)  should  become  involved  early 
in  the  system's  life  cycle,  usually  before 
program  starts  at  Milestone  I.  At  this 
time,  they  can  begin  to  develop  strategies 
for  conducting  operational  tests  OTs.  As 
test  planning  continues,  a  more-detailed 
Test  and  Evaluation  Master  Plan  (TEMP), 
Part  IV  (OT&E),  is  developed  and  the  test 
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resources  are  identified  and  scheduled. 
During  the  early  stages,  the  OTAs  structure 
an  OT&E  program  consistent  with  the  ap¬ 
proved  acquisition  strategy  for  the  system, 
identify  critical  OT  issues,  and  assess  the 
adequacy  of  candidate  systems.  As  the  pro¬ 
gram  moves  into  advanced  planning,  OT &E 
efforts  are  directed  toward  becoming  fa¬ 
miliar  with  the  system,  encouraging  inter¬ 
face  between  the  user  and  developer  and 
further  refining  the  critical  operational  is¬ 
sues  (COI).  The  OTA  test  directors,  ana¬ 
lysts  and  evaluators  design  the  OT&E  so 
that  the  data  collected  will  support  answer¬ 
ing  the  COIs.  Each  Service  has  an  indepen¬ 
dent  organization  dedicated  to  planning, 
executing  and  reporting  the  results  of  that 
Service's  OT&E  activities.  These  organiza¬ 
tions  are  the:  Army  Operational  Test  and 
Evaluation  Command  (OPTEC),  Navy  Op¬ 
erational  Test  and  Evaluation  Force 
(OPTEVFOR),  Air  Force  Operational  Test 
and  Evaluation  Center  (AFOTEC),  and 
MarineCorps  Operational  Test  and  Evalua¬ 
tion  Activity  (MCOTE  A). 

11.3.2  Test  Personnel 

Operational  testing  is  conducted  on  mate¬ 
riel  systems  with  "typical"  user  organiza¬ 
tional  units  in  a  realistic  operational  envi¬ 
ronment.  It  uses  personnel  with  the  same 
military  occupational  specialties  as  those 
who  will  operate,  maintain  and  support 
the  system  when  fielded.  Participants  are 
trained  in  the  system’s  operation  based  on 
the  Service's  operational  mission  profiles. 
Because  some  OTs  consist  of  force-on-force 
tests,  the  forces  opposing  the  tested  system 
must  also  be  trained  in  the  use  of  threat 
equipment,  tactics,  and  doctrine.  For  op¬ 
erational  testing  conducted  before  initial 
operational  test  and  evaluation  (lOT&E), 
most  system  training  is  conducted  by  the 
system’s  contractor.  For  lOT&E,  the  con¬ 
tractor  trains  the  Service  school  cadre  who 
then  train  the  participating  organizational 
units.  Once  the  system  has  entered  full-rate 


production,  the  Service  will  normally  as¬ 
sume  training  responsibilities.  Operational 
testing  often  requires  a  large  support  staff 
of  data  collectors  and  scenario  controllers 
operating  in  the  field  with  the  user  test 
forces  and  opposing  forces. 

11.4  TYPES  OF  OT&E 

Operational  Test  and  Evaluation  can  be 
subdivided  into  two  phases:  operational 
testing  performed  before  MS  III  (full-rate 
production  decision)  and  the  operational 
testing  performed  after  Milestone  (MS)  HI. 
The  Pre-MS  III  OT&E  includes  operational 
assessments  and  lOT&E.  Operational  as¬ 
sessments  begin  early  in  the  program,  fre¬ 
quently  before  program  start  (MS  I)  and 
continue  imtil  the  system  is  certified  as 
ready  for  lOT&E.  The  initial  lOT&E  is  con¬ 
ducted  just  before  the  full-rate  production 
decision.  After  MS  III,  all  subsequent  op¬ 
erational  testing  is  referred  to  as  follow-on 
operational  test  and  evaluation  (FOT&E). 
In  the  Air  Force,  if  no  research  and  develop¬ 
ment  funding  is  committed  to  a  system, 
qualification  OT&E  (QOTE)  may  be  per¬ 
formed  in  lieu  of  lOT&E.  The  Navy  uses  the 
term  "OPEVAL"  (Operational  Evaluation) 
to  define  lOT&E. 

11.4.1  Early  Operational  Assessments 

Early  operational  assessments  (EOA)  are 
conducted  primarily  to  forecast  and  evalu¬ 
ate  the  potential  operational  effectiveness 
and  suitability  of  the  weapon  system  dur¬ 
ing  development.  Early  operational  assess¬ 
ments  start  in  the  Concept  Exploration 
Phase  and  are  conducted  on  the  develop¬ 
ing  system  until  MS  II. 

11.4.1.1  Operational  Assessments 

Operational  assessments(OA)  begin  af¬ 
ter  MS  II,  when  the  OTAs  start  their  evalu¬ 
ations  of  system-level  performance.  The 
OTA  uses  any  testing  results,  modeling 
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and  simulation,  and  data  from  other 
sources  during  an  assessment.  These  data 
are  evaluated  by  the  OTA  from  an  opera¬ 
tional  point  of  view.  As  the  program  ma¬ 
tures,  these  operational  assessments  of  per¬ 
formance  requirements  are  conducted  on 
preproduction  articles  until  the  system  is 
fully  developed  and  certified  ready  for  its 
lOT&E  (OPEVAL  in  the  Navy). 

11.4.1.2  Initial  Operational 

Test  and  Evaluation  (Navy  OPEVAL) 

Initial  operational  test  and  evaluation  is  the 
final  dedicated  phase  of  OT&E  preceding  a 
full-rate  production  decision.  It  is  the  final 
evaluation  that  entails  dedicated  opera¬ 
tional  testing  of  production-representative 
test  articles  and  uses  typical  operational 
personnel  in  a  scenario  that  is  as  realistic  as 
possible  (10  U.S.C.  2399).  The  lOT&E  is 
conducted  by  an  OT&E  agency  indepen¬ 
dent  of  the  contractor,  PMO  or  developing 
agency.  The  test  has  been  described  as: 

All  operational  test  and  evaluation 
conducted  on  production  or  produc¬ 
tion  representative  articles,  to  support 
the  decision  to  proceed  beyond  low 
rate  initial  production.  It  is  conducted 
to  provide  a  valid  estimate  of  expected 
system  operational  effectiveness  and 
operational  suitability.  The  definition 
of  "OT&E"  as  spelled  out  in  congres¬ 
sional  legislation  (see  glossary)  is  gen¬ 
erally  considered  applicable  only  to 
initial  operational  test  and  evaluation 
(lOT&E). 

Further,  lOT&E  must  be  conducted 
without  system  contractor  personnel 
participation  in  any  capacity  other  than 
stipulated  in  service  wartime  tactics 
and  doctrine  as  set  forth  in  Public  Law 
99-661  by  Congress.  The  results  from 
this  test  are  evaluated  and  presented 
to  the  milestone  decision  authority  (i.e. 


MS  in,  the  decision  to  enter  full-rate 
production)  to  support  the  beyond- 
low-rate  initial  production  (LRIP)  de¬ 
cision.  This  phase  of  OT&E  addresses 
the  critical  issues  identified  in  the  Op¬ 
erational  Requirements  Document 
(ORD)andtheTEMP.IOT&Etestplans 
for  ACAT  I  and  LA  and  other  desig¬ 
nated  programs  must  be  approved  by 
the  OSD  Director,  Operational  Test 
and  Evaluation.  Service  lOT&E  test 
reports  provide  the  foundation  for  the 
DOT&E  MS  ni  Beyond  LRIP  report. 

11.4.2  Follow-On  Operational 
Test  and  Evaluation 

Follow-on  operational  test  and  evaluation 
is  conducted  after  the  MS  III  decision.  The 
tests  are  conducted  in  a  realistic  tactical 
environment  similar  to  that  used  in  lOT&E, 
but  many  test  items  may  be  used.  Normally 
FOT&E  is  conducted  using  fielded  produc¬ 
tion  systems.  Specific  objectives  of  FOT&E 
include  testing  modifications  that  are  to  be 
incorporated  into  production  systems,  com¬ 
pleting  any  deferred  or  incomplete  lOT&E, 
and  assessing  reliability  including  spares 
support.  The  tests  are  also  used  to  evaluate 
the  system  in  a  different  platform  applica¬ 
tion  for  new  tactical  applications  or  against 
new  threats. 

11.4.3  Qualification  Operational 
Test  and  Evaluation  (USAF) 

Air  Force  qualification  operational  test  and 
evaluation  may  be  performed  by  the  major 
command,  user  or  AFOTEC  and  is  con¬ 
ducted  on  minor  modifications  or  new  ap¬ 
plications  of  existing  equipment  when  no 
research  and  development  funding  is  re¬ 
quired.  An  example  of  a  program  in  which 
QOT&E  was  performed  by  the  Air  Force  is 
the  A-10  Air-to-Air  Self  Defense  Program. 
In  this  program  the  mission  of  the  A-10  was 
expanded  from  strictly  ground  support  to 
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include  an  air-to-air  defense  role.  To  ac¬ 
complish  this  the  A-10  aircraft  was  modi¬ 
fied  with  off-the-shelf  AIM-9  and  air-to- 
air  missiles;  QOT&E  was  performed  on 
the  system  to  evaluate  its  operational  ef¬ 
fectiveness  and  suitability. 

11.5  TEST  PLANNING 

Operational  test  planning  is  one  of  the 
most  important  parts  of  the  OT&E  pro¬ 
cess.  Proper  planning  facilitates  the  ac¬ 
quisition  of  data  to  support  the  determi¬ 
nation  of  the  weapon  system's  operational 
effectiveness  and  suitability.  Planning 
must  be  pursued  in  a  deliberate,  compre¬ 
hensive  and  structured  manner.  Careful 
and  complete  planning  may  not  guaran¬ 
tee  a  successful  test  program;  but  inad¬ 
equate  planning  can  result  in  significant 
test  problems,  poor  system  performance 
and  cost  overruns.  Operational  test  plan¬ 
ning  is  conducted  by  the  OTA  before 
program  start,  and  more-detailed  plan¬ 
ning  usually  starts  about  two  years  be¬ 
fore  each  operational  test  event. 

Operational  planning  can  be  divided  into 
three  phases:  early  planning,  advanced 
planning  and  detailed  planning.  Early 
planning  entails  developing  critical  op¬ 
erational  issues,  formulating  a  plan  for 
evaluations,  determining  the  concept  of 
operation,  envisioning  the  operational  en¬ 
vironment  and  developing  mission  sce¬ 
narios  and  resource  requirements.  Ad¬ 
vanced  planning  encompasses  the  deter¬ 
mination  of  the  purpose  and  scope  of 
testing  and  identification  of  measures  of 
effectiveness  (MOEs)  for  critical  issues.  It 
includes  developing  test  objectives,  es¬ 
tablishing  a  test  approach,  and  estimat¬ 
ing  test  resource  requirements.  Detailed 
planning  involves  developing  step-by- 
step  procedures  to  be  followed  as  well  as 
the  final  coordination  of  resource  require¬ 
ments  necessary  to  carry  out  OT&E. 


11.5.1  Testing  Critical 
Operational  Issues  (COI) 

Critical  operational  issues  have  been  de¬ 
scribed  as; 

A  key  operational  effectiveness  or 
operational  suitability  issue  that  must 
be  examined  in  operational  test  and 
evaluation  to  determine  the  system's 
capability  to  perform  its  mission.  A 
critical  operational  issue  is  normally 
phrased  as  a  question  to  be  answered 
in  evaluating  a  system's  operational 
effectiveness  and/or  operational 
suitability. 

One  of  the  purposes  of  OT&E  is  to  resolve 
COIs  about  the  system.  The  first  step  in  an 
OT&E  program  is  to  identify  these  critical 
issues,  some  of  which  are  explicit  in  the 
operational  requirement  document.  Ex¬ 
amples  can  be  found  in  questions  such  as: 
"How  well  does  the  system  perform  a  par¬ 
ticular  aspect  of  its  mission?"  "Can  the  sys¬ 
tem  be  supported  logistically  in  the  field?" 
Other  issues  arise  from  questions  asked 
about  system  performance  or  how  it  will 
affect  other  systems  with  which  it  must 
operate.  Critical  issues  provide  focus  and 
direction  for  the  operational  test.  Identi¬ 
fying  the  issues  is  analogous  to  the  first 
step  in  the  system  engineering  process, 
that  is,  defining  the  problem.  When  criti¬ 
cal  operational  issues  are  properly  ad¬ 
dressed,  deficiencies  in  the  system  can  be 
uncovered.  They  form  the  basis  for  a  struc¬ 
tured  technique  of  analysis  by  which  de¬ 
tailed  subobjectives  or  MOEs  can  be  es¬ 
tablished.  During  the  operational  test, 
each  subobjective  is  addressed  by  an  ac¬ 
tual  test  measurement  (measure  of  per¬ 
formance).  After  these  issues  are  identi¬ 
fied,  the  evaluation  plans  and  test  design 
are  developed  for  test  execution.  (For  more 
information,  see  the  chapter  on  Evalua¬ 
tion.) 
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11.5.2  Test  Realism 


Test  realism  for  OT&E  will  vary  directly 
with  the  degree  of  system  maturity.  Efforts 
early  in  the  acquisition  program  should 
focus  on  active  involvement  of  users  and 
operationally  oriented  environments.  Fi¬ 
delity  of  the  "combat  environment"  should 
peak  during  the  lOT&E  when  force-on- 
force  testing  of  the  production  representa¬ 
tive  system  is  conducted.  The  degree  of 
success  in  replicating  a  realistic  operational 
environment  has  a  direct  impact  on  the 
credibility  of  the  lOT&E  test  report.  Areas 
of  primary  concern  for  the  test  plaimer  can 
be  derived  from  the  legislated  definition  of 
OT&E: 

(1)  A  field  test  includes  all  of  the 
elements  normally  expected  to  be  en¬ 
countered  in  the  operational  arena, 
such  as  appropriate  size  and  type  of 
maneuver  terrain,  environmental  fac¬ 
tors,  day  / night  operations,  austere  liv¬ 
ing  conditions,  etc. 

(2)  Realistic  combat  should  be  repli¬ 
cated  using  appropriate  tactics  and 
doctrine,  representative  threat  forces 
properly  trained  in  the  employment  of 
threat  equipment,  free  play  responses 
to  test  stimulus,  stress,  "dirty"  battle 
area  (fire,  smoke,  nuclear,  biological 
and  chemical  (NBC);  electronic  coun¬ 
termeasures  (ECM),  etc.),  wartime 
tempo  to  operations,  real  time  casu¬ 
alty  assessment,  and  forces  requiring 
interoperability. 

(3)  Any  item  means  the  production 
representative  configuration  of  the 
system  at  that  point  in  time,  including 
appropriate  logistics  tail. 

(4)  Typical  military  users  are  obtained 
by  taking  a  cross  section  of  adequately 


trained  skill  levels  and  ranks  of  the 
intended  operational  force.  Selection 
of  "golden  crews"  or  the  best  of  the  best 
does  not  provide  test  data  reflecting 
the  successes  nor  problems  of  the 
"murphy  and  gang"  of  typical  units. 

In  his  book.  Operational  Test  and  Evaluation, 
Roger  Stevens  states,  "In  order  to  achieve 
realism  effectively  in  an  OT&E  program,  a 
concern  for  realism  must  pervade  the  en¬ 
tire  test  program  from  the  very  beginning 
of  test  planning  to  the  time  when  the  very 
last  test  iteration  is  run."  Realism  is  a  sig¬ 
nificant  issue  during  planning  and  execu¬ 
tion  of  OT&E  (Reference  114). 

11.5.3  Selection  of  a  Test  Concept 

An  important  step  in  the  development  of 
an  OT&E  program  is  to  develop  an  overall 
test  program  concept.  Determinations  must 
be  made  regarding  when  OT&E  will  be 
conducted  during  systems  development, 
what  testing  is  to  be  done  on  production 
equipment,  how  the  testing  will  be  evolu¬ 
tionary,  and  what  testing  will  have  to  wait 
until  all  system  capabilities  are  developed. 
This  concept  can  best  be  developed  by  con¬ 
sidering  a  number  of  aspects  such  as  test 
information  requirements,  system  availabil¬ 
ity  for  test  periods,  and  the  demonstration 
of  system  capabilities.  The  test  concept  is 
driven  by  the  acquisition  strategy  and  is  a 
road  map  used  for  planning  test  and  evalu¬ 
ation  events.  The  Director,  Operational  Test 
and  Evaluation  is  briefed  on  test  concepts 
for  oversight  programs  before  lOT &E  starts . 

11.6  TEST  EXECUTION 

An  operational  test  plan  is  only  as  good 
as  the  execution  of  that  plan.  The  execu¬ 
tion  is  the  essential  bridge  between  test 
planning  and  test  reporting.  The  test  is 
executed  through  the  OTA  test  director's 
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efforts  and  the  actions  of  the  test  team.  For 
successful  execution  of  the  OT&E  plan,  the 
test  director  must  direct  and  control  the  test 
resources  and  collect  the  data  required  for 
presentation  to  the  decision  authority.  The 
test  director  must  prepare  for  testing,  acti¬ 
vate  and  train  the  test  team,  develop  test 
procedures  and  operating  instructions,  con¬ 
trol  data  management,  create  OT&E  plan 
revisions,  and  manage  each  of  the  test  tri¬ 
als.  The  test  director's  data  management 
duties  will  encompass  collecting  raw  data, 
creating  a  data  status  matrix,  and  ensuring 
data  quality  by  processing  and  reducing, 
verifying,  filing,  storing,  retrieving  and  ana¬ 
lyzing  collected  data.  Once  all  tests  have 
been  completed  and  the  data  is  reduced 
and  analyzed,  the  results  must  be  reported. 
A  sample  test  organization  used  for  the 
Army  OT&E  of  the  improved  81mm  mor¬ 
tar  is  illustrated  in  Figure  1 1  -1 .  (In  the  Army, 
the  Deputy  Test  Director  comes  from  the 
OTA  and  controls  the  daily  OT  activity.) 

11.7  TEST  REPORTING 

The  lOT&E  test  report  is  a  very  important 
document.  It  must  communicate  the  re¬ 
sults  of  completed  tests  to  decision  authori¬ 
ties  in  a  timely,  factual,  concise,  compre¬ 
hensive  and  accurate  manner.  The  report 
must  present  a  balanced  view  of  the  weapon 
system's  successes  and  problems  during 
testing,  illuminating  both  the  positive  as¬ 
pects  and  system  deficiencies  discovered. 
Analysis  of  test  data  and  their  evaluation 
may  be  in  one  report  (USAF,  USN)  or  in 
separate  documents  (USA,  USMC). 

There  are  four  types  of  reports  most  fre¬ 
quently  used  in  reporting  OT&E  results. 
These  include  status,  interim,  quick-look 


and  final  reports.  The  status  report  gives 
periodic  updates  (e.g.,  monthly,  quarterly) 
and  reports  recent  test  findings  (discreet 
events  such  as  missile  firings).  The  interim 
report  provides  a  summary  of  the  cumula¬ 
tive  test  results  to  date  when  there  is  an 
extended  period  of  testing.  The  quick-look 
reports  provide  preliminary  test  results, 
are  usually  prepared  immediately  after  a 
test  event  (less  than  7  days)  and  have  been 
used  to  support  program  decision  mile¬ 
stones.  The  final  test  and  evaluation  report 
(Air  Force,  Navy)  or  independent  evalua¬ 
tion  report  (Army,  Marine)  presents  the 
conclusions  and  recommendations  includ¬ 
ing  all  supporting  data  and  covering  the 
entire  lOT&E  program. 

11.8  SUMMARY 

The  purpose  of  OT&E  is  to  assess  opera¬ 
tional  effectiveness  and  suitability  at  each 
stage  in  the  acquisition  process.  Opera¬ 
tional  effectiveness  is  a  measure  of  the  con¬ 
tribution  of  the  system  to  mission  accom¬ 
plishment  imder  actual  conditions  of  em¬ 
ployment.  Operational  suitability  is  a  mea¬ 
sure  of  the  maintainability  and  reliability 
of  the  system;  the  effort  and  level  of  train¬ 
ing  required  to  maintain,  support  and  op¬ 
erate  it;  and  any  unique  logistic  or  training 
requirements  it  may  have.  The  OT&E  may 
provide  information  on  tactics,  doctrine, 
organization  and  persoimel  requirements 
and  may  be  used  to  assist  in  the  prepara¬ 
tion  of  operating  and  maintenance  instruc¬ 
tions  and  other  publications.  One  of  the 
most  important  aspects  is  that  OT&E  pro¬ 
vides  an  independent  evaluation  of  the 
degree  of  progress  made  toward  satisfying 
the  user's  requirements  during  the  system 
development  process. 
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Figure  11-1.  Organizational  Breakdown  of  the  I-81mm  Mortar  Operational  Test  Directorate 


OT&E  TO  SUPPORT 
MILESTONE  DECISIONS 


12.1  INTRODUCTION 

Mindful  of  principles  of  objectivity  and 
impartial  evaluation,  operational  test  and 
evaluation  (OT&E)  may  be  conducted  be¬ 
fore  each  major  milestone  (MS)  review  to 
provide  the  decision  authority  with  the 
latest  results  from  testing  of  critical  opera¬ 
tional  issues.  The  philosophy  of  OT&E  has 
been  related  to  three  terms  —  adequacy, 
quality  and  credibility: 

Adequacy  —  The  amount  of  data  and 
realism  of  test  conditions  must  be  sufficient 
to  support  the  evaluation  of  the  critical 
operational  issues. 

Quality  —  The  test  planning,  control  of 
test  events,  and  treatment  of  data  must 
provide  clear  and  accurate  test  reports. 

Credibility — The  conduct  of  the  test  and 
data  handling  must  be  separated  from  ex¬ 
ternal  influence  and  personal  biases. 

Operational  testing  is  conducted  to  pro¬ 
vide  information  to  support  department  of 
Defense  (DoD)  executive-level  manage¬ 
ment  decisions  on  major  acquisition  pro¬ 
grams.  Operational  test  and  evaluation  is 
accomplished  using  a  test  cycle  of  succes¬ 
sive  actions  and  documents.  During  the 
early  stages  of  the  program,  the  process  is 
informal  and  modified  as  necessary.  As 
programs  mature,  documentation  for  ma¬ 
jor  systems  and  those  designated  by  the 
Director,  Operational  Test  and  Evaluation 
(DOT&E)  for  oversight  must  be  sent  to  the 


Office  of  the  Secretary  of  Defense  (OSD)  for 
approval  before  the  testing  can  be  con¬ 
ducted  or  the  systems  can  be  cleared  to 
proceed  beyond  low  rate  initial  production 
(BLRIP).  Figure  12-1  illustrates  how  OT&E 
relates  to  the  acquisition  process. 

12.2  OT&E  DURING  CONCEPT 
EXPLORATION-  PHASE  0 
(MS  0  to  MS  I) 

The  OT&E  conducted  during  the  Con¬ 
cept  Exploration  (CE)  Phase  is  an  early 
operational  assessment  (EOA)  focused  on 
investigating  the  deficiencies  identified 
during  the  mission  area  analysis.  Opera¬ 
tional  testers  participate  in  these  evalua¬ 
tions  to  validate  the  OT&E  requirements 
for  future  testing  and  to  identify  issues 
and  criteria  that  can  only  be  resolved 
through  OT&E  to  initiate  early  test  re¬ 
source  planning. 

Before  MS  I,  the  OT&E  objectives  are  to 
assist  in  evaluating  alternative  concepts  to 
solve  the  mission  area  deficiencies  and  to 
assess  the  operational  impact  of  the  sys¬ 
tem.  This  early  assessment  also  provides 
data  to  support  a  decision  on  whether  to 
enter  the  Program  Definition  and  Risk  Re¬ 
duction  Phase.  The  OT&E  conducted  dur¬ 
ing  Phase  0  supports  developing  estimates 
of: 

(1)  The  military  need  for  the  proposed 
system; 
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(2)  A  demonstration  that  there  is  a  sound 
physical  basis  for  a  new  system; 

(3)  An  analysis  of  concepts,  based  on 
demonstrated  physical  phenomena,  for 
satisfying  the  military  need; 

(4)  The  system's  affordability  and  life- 
cycle  cost; 

(5)  The  ability  of  a  modification  to  an 
existing  U.S.  or  allied  system  to  provide 
needed  capability; 

(6)  An  operational  utility  assessment; 

(7)  An  impact  of  the  system  on  the  force 
structure. 

At  MS  I,  there  is  normally  no  hardware 
available  for  the  operational  tester.  There¬ 
fore,  the  early  operational  assessment  is 
conducted  from  surrogate  test  and  experi¬ 
ment  data,  breadboard  models,  factory 
user  trials,  mock-up /simulators,  model¬ 
ing/simulation,  and  user  demonstrations 
(Figure  12-2).  This  makes  early  assess¬ 
ments  difficult,  and  some  areas  cannot  be 
covered  in-depth.  However,  these  assess¬ 
ments  provide  vital  introductory  infor¬ 
mation  on  the  system's  potential  opera¬ 
tional  utility. 

The  OT&E  products  from  this  phase  of 
testing  include  the  information  provided 
to  the  decision  authority,  data  collected 
for  further  evaluation,  input  to  the  Test 
and  Evaluation  Master  Plan  (TEMP)  and 
early  test  and  evaluation  (T&E)  planning. 
Special  logistics  problems,  program  ob¬ 
jectives,  program  plans,  performance 
parameters  and  acquisition  strategy  are 
areas  of  primary  influence  to  the  opera¬ 
tional  tester  during  this  phase  and  must 
be  carefully  evaluated  to  project  the 


system's  operational  effectiveness  and 
suitability. 

12.3  OT&E  DURING 
PROGRAM  DEFINITION 
AND  RISK  REDUCTION-  PHASE  I 
(MS  I  to  MS  II) 

Combined  development  test  (DT)/OT&E 
or  an  early  operational  assessment  during 
Phase  I,  is  conducted  to  support  the  MS  II 
decision  regarding  a  system's  readiness  to 
move  into  the  Engineering  and  Manufac¬ 
turing  (EMD)  Phase.  In  all  cases,  appropri¬ 
ate  T&E  must  be  conducted  before  the  MS 
II  decision,  thereby  providing  data  for  iden¬ 
tification  of  risk  before  more  resources  are 
committed.  As  appropriate,  low  rate  initial 
production  (LRIP)  may  be  approved  at  MS 
II  to  verify  production  capability  and  to 
provide  test  resources  needed  to  conduct 
interoperability,  live  fire,  or  operational 
testing. 

12,3.1  Objectives  of  Early 
Operational  Assessments 

Early  operational  assessments  are  con¬ 
ducted  to  facilitate  identification  of  the  best 
design,  indicate  the  risk  level  of  perfor¬ 
mance  for  this  phase  of  the  development, 
examine  operational  asp)ects  of  the  system's 
development,  and  estimate  potential  op¬ 
erational  effectiveness  and  suitability.  Ad¬ 
ditionally,  an  analysis  of  the  planning  for 
transition  from  development  to  produc¬ 
tion  is  initiated.  Early  operational  assess¬ 
ments  supporting  the  MS  II  decision  are 
intended  to: 

(1)  Assess  the  potential  of  the  new 
system  in  relation  to  existing  capabilities; 

(2)  Assess  system  effectiveness  and 
suitability  so  that  affordability  can  be 
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evaluated  for  program  cost  versus  mili¬ 
tary  utility; 

(3)  Assess  the  adequacy  of  the  concept 
for  employment,  supportability  and  orga¬ 
nization;  doctrinal,  tactical  and  training  re¬ 
quirements;  and  related  critical  issues; 

(4)  Estimate  the  need  for  the  selected 
systems  in  consideration  of  the  threat  and 
system  alternatives  based  on  military  util¬ 
ity; 

(5)  Assess  the  validity  of  the  operational 
concept; 

(6)  List  the  key  risk  areas  and  critical 
operational  issues  that  need  to  be  resolved 
before  EMD  is  initiated; 

(7)  Assess  the  need  for  LRIP  of  hardware 
to  support  initial  operational  test  and  evalu¬ 
ation  (lOT&E)  prior  to  the  full-rate  produc¬ 
tion  decision; 

(8)  Provide  data  to  support  test  planning 
for  the  EMD  Phase. 

During  this  phase,  OT&E  may  be  conducted 
on  brassboard  configurations,  experimen¬ 
tal  prototypes  or  advanced  development 
prototypes.  Dedicated  test  time  may  be 
made  available  for  the  operational  tester. 
However,  the  OT&E  assessments  may  also 
make  use  of  many  other  additional  data 
sources.  Examples  of  additional  sources 
often  used  by  the  Army  during  this  phase 
include:  concept  evaluation  program  tests, 
innovative  testing,  force  development  tests 
and  experimentation  (FDT&E),  source  se¬ 
lection  tests,  user  participation  in  develop¬ 
ment  test  and  evaluation  (DT&E)  and  op¬ 
erational  feasibility  tests.  The  results  from 
this  testing,  analysis  and  evaluation  are 
documented  in  the  Early  Operational  As¬ 
sessment  (EOA)  or  end  of  phase  OT&E 
report.  These  data,  along  with  the  mission 


needs  and  requirements  documentation 
and  TEMP,  assist  in  the  review  of  perfor¬ 
mance  for  the  MS  II  decision. 

12.4  OT&E  DURING 
ENGINEERING  AND 
MANUFACTURING 
DEVELOPMENT  -  PHASE  II 
(MS  II  to  MS  III) 

The  lOT&E  and  operational  assessments 
during  the  EMD  Phase  are  conducted  on 
engineering  development  models,  produc¬ 
tion  representative  or  production  systems. 
These  operational  evaluations  estimate  the 
operational  effectiveness  and  suitability  and 
provide  data  on  whether  the  system  meets 
minimum  operational  thresholds.  Just  be¬ 
fore  the  full-rate  production  MS  HI  deci¬ 
sion,  the  dedicated  T&E  is  conducted  on 
equipment  that  has  been  formally  certified 
by  the  program  manager  as  being  ready  for 
the  "final  OT&E."  This  dedicated  lOT&E  is 
conducted  in  a  test  environment  as  opera¬ 
tionally  realistic  as  possible. 

12.4.1  OT&E  Objectives 

The  OA/IOT&E  conducted  during  EMD, 
is  characterized  by  testing  performed  by 
user  organizations  in  a  field  exercise  to 
examine  the  organization  and  doctrine,  in¬ 
tegrated  logistics  support,  threat,  commu¬ 
nications,  command  and  control,  and  tac¬ 
tics  associated  with  the  operational  em¬ 
ployment  of  the  unit  during  tactical  opera¬ 
tions.  This  includes  estimates  which: 

(1)  Assess  operational  effectiveness  and 
suitability; 

(2)  Assess  the  survivability  of  the  sys¬ 
tem; 

(3)  Assess  the  systems  reliability,  main¬ 
tainability  and  plans  for  integrated  logis¬ 
tics  support; 


12-5 


(4)  Evaluate  manpower,  personnel,  train¬ 
ing  and  safety  requirements; 

(5)  Validate  organizational  and  employ¬ 
ment  concepts; 

(6)  Determine  training  and  logistics  re¬ 
quirements  deficiencies; 

(7)  Assess  the  system's  readiness  to  enter 
full-rate  production. 

12.5  OT&E  DURING 
PRODUCTION ,  FIELDING/ 
DEPLOYMENT,  AND  OPERATIONAL 
SUPPORT-  PHASE  III 
(MS  III  to  AND  BEYOND) 

After  the  MS  III  decision,  the  emphasis 
shifts  towards  procuring  production  quan¬ 
tities,  repairing  hardware  deficiencies, 
managing  changes,  and  phasing  in  full  lo¬ 
gistics  support.  During  initial  deployment 
of  the  system,  the  OT&E  agency  and/or  the 
user  may  perform  follow-on  operational 
test  and  evaluation  (FOT&E)  to  refine  the 
effectiveness  and  suitability  estimates  made 
during  earlier  OT&E,  assess  performance 
not  evaluated  during  lOT&E,  evaluate  new 
tactics  and  doctrine,  and  assess  the  impacts 
of  system  modifications  or  upgrades. 

The  FOT&E  is  performed  with  produc¬ 
tion  articles  in  operational  organizations. 
It  is  normally  funded  with  operation  and 
maintenance  (O&M)  funds.  The  first 
FOT&E  conducted  during  this  phase  may 
be  used  to: 

(1)  Ensure  that  the  production  system 
performs  as  well  as  reported  at  the  MS  III 
review; 

(2)  Demonstrate  expected  performance 
and  reliability  improvements; 


(3)  Ensure  that  the  correction  of  deficien¬ 
cies  identified  during  earlier  testing  have 
been  completed; 

(4)  Evaluate  performance  not  tested  dur¬ 
ing  lOT&E. 

Additional  objectives  of  FOT&E  are  to 
validate  the  operational  effectiveness  and 
suitability  of  a  modified  system  during 
an  operational  assessment  of  the  system 
in  new  environments.  The  FOT&E  may 
look  at  different  platform  applications, 
new  tactical  applications  or  the  impact  of 
new  threats. 

12.5.1  FOT&E  of  Logistic 
Support  Systems 

The  testing  objectives  to  evaluate 
postproduction  logistics  readiness  and  sup¬ 
port  are  to: 

(1)  Assess  the  logistics  readiness  and 
sustainability; 

(2)  Evaluate  the  weapon  support  objec¬ 
tives; 

(3)  Assess  the  implementation  of  logis¬ 
tics  support  planning; 

(4)  Evaluate  the  capability  of  the  logis¬ 
tics  support  activities; 

(5)  Determine  the  disposition  of  dis¬ 
placed  equipment; 

(6)  Evaluate  the  affordability  and  life- 
cycle  cost  of  the  system. 

12.6  SUMMARY 

Operational  test  and  evaluation  is  that  T&E 
(operational  assessments,  lOT&E  or 
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FOT&E)  conducted  to  estimate  a  system's 
operational  effectiveness  and  operational 
suitability.  They  will  identify  needed 
modifications;  provide  information  on 
tactics,  doctrine,  organizations  and  per¬ 
sonnel  requirements;  and  evaluate  the 


system's  logistic  supportability.  The  ac¬ 
quisition  program  structure  should  in¬ 
clude  operational  assessments  or  evalua¬ 
tions  beginning  early  in  the  development 
cycle  and  continuing  throughout  the 
system's  life  cycle. 
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MODULE 

Test  and  Evaluation 
Planning 


Many  program  managers  face  several  T&E 
issues  that  must  be  resolved  to  get  their 
particular  weapon  system  tested  and  ulti¬ 
mately  fielded.  These  issues  may  include 
modeling  and  simulation  support,  com¬ 
bined  and  concurrent  testing,  test  resources, 
survivability  and  lethality  testing,  multi- 
Service  testing,  or  international  T&E.  Each 
issue  presents  a  unique  set  of  challenges  for 
the  program  manager  when  he/ she  devel¬ 
ops  the  integrated  strategy  for  the  T&E 
program. 


EVALUATION 


13.1  INTRODUCTION 

This  chapter  describes  the  evaluation  por¬ 
tion  of  the  test  and  evaluation  (T&E)  pro¬ 
cess.  It  stresses  the  importance  of  establish¬ 
ing  and  maintaining  a  clear  audit  trail  from 
system  requirements  through  critical  is¬ 
sues,  evaluation  criteria,  test  objectives  and 
measures  of  effectiveness  to  the  evalua¬ 
tion.  The  importance  of  the  use  of  data  from 
all  sources  is  discussed  as  are  the  differ¬ 
ences  in  approaches  to  evaluating  technical 
and  operational  data. 

13.2  DIFFERENCE  BETWEEN 
•TEST”  AND  "EVALUATION” 

The  following  distinction  has  been  made 
between  the  functions  of  "test"  and  "evalu¬ 
ation:" 

While  the  terms  "test"  and  "evalua¬ 
tion"  are  most  often  found  together, 
they  actually  denote  clearly  distin¬ 
guishable  functions  in  the  RDT&E  [re¬ 
search,  development,  test  and  evalua¬ 
tion]  process. 

"Test"  denotes  the  actual  testing  of  hard¬ 
ware/software — models,  prototypes,  pro¬ 
duction  equipment,  computer  programs 
—  to  obtain  data,  both  quantitative  and 
qualitative,  relevant  to  developing  new 
capabilities,  managing  the  process,  or  mak¬ 
ing  decisions  on  the  allocation  of  resources. 

"Evaluation"  denotes  the  process  whereby 
data  are  logically  assembled,  analyzed,  and 


compared  to  expected  performance  to  aid 
in  making  systematic  decisions. 

To  summarize,  evaluation  is  the  process  for 
review  and  analysis  of  qualitative  or  quan¬ 
titative  data  obtained  from  design  review, 
hardware  inspection,  modeling  and  simu¬ 
lation,  testing  or  operational  usage  of  equip¬ 
ment. 

13.3  THE  EVALUATION  PROCESS 

The  evaluation  process  requires  a  broad 
analytical  approach  with  careful  focus  on 
the  development  of  an  overall  (T&E  plan 
that  will  provide  timely  answers  to  critical 
issues  and  questions  required  by  decision 
authorities  throughout  all  the  acquisition 
phases.  (Table  13-1)  Evaluations  should 
focus  on  key  performance  parameters;  i.e., 
"that  capability  or  characteristic  so  signifi¬ 
cant  that  failure  to  meet  the  threshold  can 
be  cause  for  the  concept  or  system  selection 
to  be  reevaluated  or  the  program  to  be 
reassessed  or  terminated"  Department  of 
Defense  (DoD)  5000.2-R). 

A  functional  block  diagram  of  a  generic 
(i.e.,  not  Service-specific)  evaluation  pro¬ 
cess  is  shown  in  Figure  13-1.  The  process 
begins  with  the  identification  of  a  defi¬ 
ciency  or  need  and  the  documentation  of 
an  operational  requirement.  It  continues 
with  the  identification  of  critical  issues 
that  must  be  addressed  to  determine  the 
degree  to  which  the  system  meets  user 


Table  13-1.  Sample  Evaluation  Plan 
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Figure  13-1.  Functional  Block  Diagram  of  the  Evaluation  Process 


requirements.  Objectives  and  thresholds 
must  then  be  established  to  define  required 
performance  or  supportability  parameters 
and  to  evaluate  progress  in  reaching  them. 
Test  and  evaluation  analysts  then  decom¬ 
pose  the  issues  into  measurable  test  ele¬ 
ments,  conduct  the  necessary  testing,  re¬ 
view  and  analyze  the  test  data,  weigh  the 
test  results  against  the  evaluation  criteria, 
and  prepare  an  evaluation  report  for  the 
decision  authorities. 

13.4  ISSUES  AND  CRITERIA 

Issues  are  questions  regarding  a  system 
that  require  answers  during  the  acquisition 
process.  Those  answers  may  be  needed  to 
aid  in  the  development  of  an  acquisition 
strategy,  to  refine  performance  require¬ 
ments  and  designs  or  to  support  milestone 
decision  reviews.  Evaluation  criteria  are 
the  standards  by  which  accomplishments 
of  required  technical  and  operational  effec¬ 
tiveness  and/or  suitability  characteristics 
or  resolution  of  operational  issues  may  be 
assessed.  The  evaluation  program  may  be 
constructed  using  a  structured  approach 
identifying  each  issue. 

(1)  Issue — a  statement  of  the  question  to 
be  answered; 

(2)  Scope  —  detailed  conditions  and 
range  of  conditions  that  will  guide  the  T&E 
process  for  this  issue; 

(3)  Criteria —  quantitative  or  qualitative 
standards  that  will  answer  the  issue; 

(4)  Rationale  —  full  justification  to  sup¬ 
port  the  selected  criteria. 

13.4.1  Key  Performance  Parameters/ 
Critical  Issues 

Key  performance  parameters  often  can 
support  the  development  of  a  hierarchy  of 
critical  issues  and  less  significant  issues. 


Critical  issues  are  those  questions  relating 
to  a  system's  operational,  technical,  sup¬ 
port  or  other  capability.  These  issues  must 
be  answered  before  the  system's  overall 
worth  can  be  estimated  /evaluated,  and  they 
are  of  primary  importance  to  the  decision 
authority  in  allowing  the  system  to  ad¬ 
vance  to  the  next  acquisition  phase.  Critical 
issues  in  the  Test  and  Evaluation  Master 
Plan  (TEMP)  may  be  derived  from  the  key 
performance  parameters  found  in  the  op¬ 
erational  requirement  document  (ORD). 
The  system  requirements  and  baseline 
documentation  will  provide  many  of  the 
performance  parameters  required  to  de¬ 
velop  the  hierarchy  of  issues. 

13.4.2  Evaluation  Issues 

Evaluation  issues  are  those  addressed  in 
technical  or  operational  evaluations  dur¬ 
ing  the  acquisition  process.  Evaluation 
issues  can  be  separated  into  technical  or 
operational  issues  and  addressed  in 
theTEMP. 

Technical  issues  primarily  concern  techni¬ 
cal  parameters  or  characteristics  and  engi¬ 
neering  specifications  normally  assessed 
in  development  testing.  Operational  issues 
concern  effectiveness  and  suitability  char¬ 
acteristics  for  functions  to  be  performed  by 
equipment/personnel.  They  address  the 
system's  operational  performance  when 
examined  in  a  realistic  operational  mission 
environment.  Evaluation  issues  are  an¬ 
swered  by  whatever  means  necessary 
(analysis /survey,  modeling,  simulation, 
inspection,  demonstration  or  testing)  to 
resolve  the  issue.  Issues  requiring  test  data 
are  further  referred  to  as  test  issues. 

13.4.3  Test  Issues 

Test  issues  are  a  subset  of  evaluation  issues 
and  address  areas  of  uncertainty  that  re¬ 
quire  test  data  to  resolve  the  issue  ad¬ 
equately.  Test  issues  may  be  partitioned 
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into  technical  issues  that  are  addressed  by 
the  development  test  and  evaluation 
(DT&E)  community  (contractor  and  gov¬ 
ernment)  and  operational  issues  that  are 
addressed  by  the  operational  test  and  evalu¬ 
ation  (OT&E)  community.  Test  issues  may 
be  divided  into  critical  and  noncritical  cat¬ 
egories.  All  critical  T&E  issues,  objectives, 
methodologies  and  evaluation  criteria 
should  be  defined  during  the  initial  estab¬ 
lishment  of  an  acquisition  program.  Criti¬ 
cal  issues  are  documented  in  the  TEMP. 
These  evaluation  issues  serve  to  define  the 
testing  required  for  each  phase  of  the  ac¬ 
quisition  process  and  serve  as  the  structure 
to  guide  the  testing  program  so  these  data 
maybe  compared  against  performance  cri¬ 
teria. 

13.4.4  Criteria 

Criteria  are  statements  of  a  system's  re¬ 
quired  technical  performance  and  opera¬ 
tional  effectiveness,  suitability  and  sup- 
portability.  Criteria  are  often  expressed  as 
"objectives  and  thresholds."  (Some  Services, 
however,  specify  performance  and  sup- 
portability  requirements  exclusively  in 
terms  of  thresholds  and  avoid  the  use  of  the 
concept  of  objectives.)  These  performance 
measurements  provide  the  basis  for  col¬ 
lecting  data  used  to  evaluate/answer  test 
issues. 

Criteria  must  be  unambiguous  and  assess¬ 
able  whether  stated  qualitatively  or  quan¬ 
titatively.  They  may  compare  the  mission 
performance  of  the  new  system  to  the  one 
being  replaced,  compare  the  new  system  to 
a  predetermined  standard,  or  compare 
mission  performance  results  using  the  new 
system  to  not  having  the  system.  Criteria 
are  the  final  values  deemed  necessary  by 
the  user.  As  such,  they  should  be  devel¬ 
oped  in  close  coordination  with  the  system 
user,  other  testers  and  specialists  in  all  other 
areas  of  operational  effectiveness  and 


suitability.  These  values  may  be  changed 
as  systems  develop  and  associated  testing 
and  evaluation  proceed.  Every  issue  should 
have  at  least  one  criteria  that  is  a  concise 
measure  of  the  function.  Values  must  be 
realistic  and  achievable  within  the  state  of 
the  art  of  engineering  technology.  A  quan¬ 
titative  or  qualitative  criterion  should  have 
a  clear  definition,  free  of  ambiguous  or 
imprecise  terminology,  such  as  "adequate," 
"sufficient"  or  "acceptable." 

13.4.4.1  Test  of  Thresholds 
and  Objectives 

An  ORD  threshold  performance  param¬ 
eter  lists  a  minimally  acceptable  require¬ 
ment  or  a  minimally  acceptable  level  of 
performance  required  by  a  test  article  or 
system  to  provide  a  system  capability  that 
will  satisfy  the  validated  mission  need. 
Thresholds  are  stated  quantitatively  when¬ 
ever  possible.  Specification  of  minimally 
acceptable  performance  in  measurable  pa¬ 
rameters  is  essential  to  selecting  appropri¬ 
ate  measures  of  effectiveness,  which,  in 
turn,  heavily  influence  test  design.  Thresh¬ 
olds  are  of  value  only  when  they  are  test¬ 
able;  i.e.,  actual  performance  can  be  mea¬ 
sured  against  them.  The  function  of  T&E  is 
to  verify  the  attainment  of  required  thresh¬ 
olds. 

Objectives  are  levels  of  performance  (es¬ 
tablished  by  the  user)  above  the  threshold 
that,  if  achieved,  will  provide  measurable 
benefits  of  additional  operational  capabil¬ 
ity,  operations  and  support.  Objectives  are 
not  normally  addressed  by  the  operational 
tester,  whose  primary  concern  is  the  re¬ 
quirement. 

Going  into  Milestone  II,  thresholds  and 
objectives  are  expanded  along  with  the  iden¬ 
tification  of  more-detailed  and  refined  per¬ 
formance  capabilities  and  characteristics 
resulting  from  trade-off  studies  and  testing 
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conducted  during  the  Program  Definition 
and  Risk  Phase.  Along  with  the  ORD,  they 
should  remain  relatively  stable  through 
production. 

13.5  MEASURES  OF 
EFFECTIVENESS 

Requirements,  thresholds  and  objectives 
established  in  early  program  documenta¬ 
tion  form  the  basis  for  evaluation  criteria. 
If  program  documentation  is  incomplete, 
the  tester  may  have  to  develop  evalua¬ 
tion  criteria  in  the  absence  of  specific 
requirements.  Evaluation  criteria  are  as¬ 
sociated  with  objectives,  sub-objectives 
and  measures  of  effectiveness  (MOEs) 
(Sometimes  partitioned  into  MOEs  and 
measures  of  suitability).  For  example,  an 
MOE  (e.g.,  airspeed)  may  have  an  associ¬ 
ated  evaluation  criterion  (e.g.,  450  knots) 
against  which  the  actual  performance 
(e.g.,  425  knots)  is  compared  to  arrive  at  a 
rating.  An  MOE  of  a  system  is  a  param¬ 
eter  that  evaluates  the  capacity  of  the 
system  to  accomplish  its  assigned  mis¬ 
sions  under  a  given  set  of  conditions. 
They  are  important  because  they  deter¬ 
mine  how  test  results  will  be  judged;  and, 
since  test  planning  is  directed  toward 
obtaining  these  measures,  it  is  important 
that  they  be  defined  early.  Generally,  the 
resolution  of  each  critical  issue  is  in  terms 
of  the  evaluation  of  some  MOE.  In  this 
case,  the  operating,  implementing,  and 
supporting  commands  must  agree  with 
the  criteria  before  the  test  organization 
makes  use  of  them  in  assessing  test  re¬ 
sults.  Ensuring  that  MOEs  can  be  related 
to  the  user's  operational  requirements  is 
an  important  consideration  when  identi¬ 
fying  and  establishing  evaluation  crite¬ 
ria.  Testers  must  ensure  that  evaluation 
criteria  and  MOEs  are  updated  if  require¬ 
ments  change.  Measures  of  effectiveness 
should  be  so  specific  that  the  system's 
effectiveness  during  developmental  and 


operational  testing  can  be  assessed  using 
some  of  the  same  effectiveness  criteria  as  the 
Analysis  of  Alternatives  (DoD  5()()0.2-R). 

13.6  EVALUATION  PLANNING 

13.6.1  Evaluation  Planning  Techniques 

Evaluation  planning  is  an  iterative  process 
that  requires  formal  and  informal  analyses 
of  system  operation  (e.g.,  threat  environ¬ 
ment,  system  design,  tactics  and 
interoperability) .  Techniques  that  have  been 
proven  effective  in  evaluation  planning 
include:  process  analysis,  design  or  engi¬ 
neering  analysis,  matrix  analysis  and  den¬ 
dritic  analysis  (Reference  61). 

13.6.1.1  Process 
AnalysisT  echni  ques 

Process  analysis  techniques  consist  of  think¬ 
ing  through  how  the  system  will  be  used  in 
a  variety  of  environments,  threats,  mis¬ 
sions  and  scenarios  in  order  to  understand 
the  events,  actions,  situations  and  results 
that  are  expected  to  occur.  This  technique 
aids  in  the  identification  and  clarification 
of  appropriate  MOEs,  test  conditions  and 
data  requirements. 

13.6.1.2  Design/Engineering 
Analysis  Techniques 

Design  or  engineering  analysis  techniques 
are  used  to  examine  all  mechanical  or  func¬ 
tional  operations  that  the  system  has  been 
designed  to  perform.  These  techniques  in¬ 
volve  a  systematic  exploration  of  the 
system's  hardware  and  software  compo¬ 
nents,  purpose,  performance  bounds,  man¬ 
power  and  personnel  considerations, 
known  problem  areas  and  impact  on  other 
components.  Exploration  of  the  way  a  sys¬ 
tem  operates  compared  to  intended  perfor¬ 
mance  functions  often  identifies  issues, 
MOEs,  specific  data,  test  events  and  re¬ 
quired  instrumentation. 
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13.6.1.3  Matrix  Analysis 
Techniques 

Matrix  analysis  techniques  are  useful  for 
analyzing  any  situation  where  two  classi¬ 
fications  must  be  cross-referenced.  For 
example,  a  matrix  of  "types  of  data" 
versus"means  of  data  collection"  can  re¬ 
veal  not  only  types  of  data  having  no 
planned  means  of  collection  but  also  re¬ 
dundant  or  backup  collection  systems. 
Matrix  techniques  are  useful  as  check¬ 
lists,  as  organizational  tools  or  as  a  way  of 
identifying  and  characterizing  problem 
areas.  Matrix  techniques  are  effective  for 
tracing  a  system's  operational  requirements 
through  contractual  specification  docu¬ 
ments,  issues  and  criteria  to  sources  of  indi¬ 
vidual  data  or  specific  test  events. 

13.6.1.4  Dendritic  Analysis 
Techniques 

Dendritic  analysis  techniques  are  an  effec¬ 
tive  way  of  decomposing  critical  issues  to 
the  point  where  actual  data  requirements 
and  test  measurements  can  be  identified.  In 
these  techniques,  issues  are  successively 
broken  down  into  objectives,  MOEs,  mea¬ 
sures  of  performance  and  data  require¬ 
ments  in  a  root-like  structure  as  depicted  in 
Figure  13-2.  In  this  approach,  objectives  are 
used  to  clearly  express  the  broad  aspects  of 
T&E  related  to  the  critical  issues  and  the 
overall  purpose  of  the  test.  Measures  of 
effectiveness  are  developed  as  subsets  of 
the  objectives  and  are  designed  to  treat 
specific  and  addressable  parts  of  the  objec¬ 
tives.  Each  MOE  is  traceable  as  a  direct 
contributor  to  one  objective  and,  through 
it,  is  identifiable  as  a  direct  contributor  to 
addressing  one  or  more  critical  issues  (Ref¬ 
erence  83).  Each  test  objective  and  MOE  is 
also  Hnked  to  one  or  more  measures  of 
performance  (quantitative  or  qualitative 
measures  of  system  performance  under 
specified  conditions)  that,  in  turn,  are  tied 


to  specific  data  elements.  The  dendritic 
approach  has  become  a  standard  military 
planning  technique. 

13.6.2  Sources  of  Data 

As  evaluation  and  analysis  planning  ma¬ 
tures,  focus  turns  toward  identifying  data 
sources  as  a  means  for  obtaining  each  data 
element.  Initial  identification  tends  to  be 
generic  such  as:  engineering  study,  simula¬ 
tion,  modeling  or  contractor  test.  Later  iden¬ 
tification  reflects  specific  studies,  models 
and/or  tests.  A  data  source  matrix  is  a 
useful  planning  tool  to  show  where  data 
are  expected  to  be  obtained  during  the  T&E 
of  the  system. 

There  are  many  sources  of  data  that  can 
contribute  to  the  evaluation.  Principal 
sources  include:  studies  and  analyses,  mod¬ 
els,  simulations,  war  games,  contractor  test¬ 
ing,  DT,  OT  and  comparable  systems. 

13.7  EVALUATING  DEVELOPMENT 
AND  OPERATIONAL  TESTS 

Technical  and  operational  evaluations  em¬ 
ploy  different  techniques  and  have  differ¬ 
ent  evaluation  criteria.  Development  test 
and  evaluation  is  often  considered  techni¬ 
cal  evaluation  while  OT&E  addresses  the 
operational  aspects  of  a  system.  Technical 
evaluation  deals  primarily  with  instru¬ 
mented  tests  and  statistically  valid  data. 
An  operational  evaluation  deals  with  op¬ 
erational  realism  and  the  combat  uncer¬ 
tainties  (Reference  76).  Development  test 
and  evaluation  uses  technical  criteria  for 
evaluating  system  performance.  These  cri¬ 
teria  are  usually  parameters  that  can  be 
measured  during  controlled  DT&E  tests. 
They  are  particularly  important  to  the  de¬ 
veloping  organization  and  the  contractor 
but  are  of  less  interest  to  the  independent 
operational  tester.  The  operational  tester 
focuses  on  issues  such  as  demonstrating 
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Figure  13-2.  Dendritic  Approach  to  Test  and  Evaluation 


target  acquisition  at  useful  ranges,  air  su¬ 
periority  in  combat,  or  the  probability  of 
accomplishing  a  given  mission.  For  ex¬ 
ample,  during  DT&E,  firing  may  be  con¬ 
ducted  on  a  round-by-round  basis  with 
each  shot  designed  to  test  an  individual 
specification  or  parameter  with  other  pa¬ 
rameters  held  constant.  Such  testing  is  de¬ 
signed  to  measure  the  technical  perfor¬ 
mance  of  the  system.  In  contrast,  in  OT&E 
proper  technical  performance  regarding 
individual  specifications/parameters  is  de- 
emphasized  and  the  environment  is  less 
controlled.  The  OT&E  authority  must  as¬ 
sess  whether,  given  this  technical  perfor¬ 
mance,  the  weapon  system  is  operationally 
effective  and  operationally  suitable  when 
employed  under  realistic  combat  (with 
opposing  force)  and  environmental  condi¬ 
tions  by  typical  personnel. 

The  emphasis  in  development  test  (DT)  is 
strictly  on  the  use  of  quantitative  data  to 
verify  attainment  of  technical  specifications. 
Quantitative  data  are  usually  analyzed  us¬ 
ing  some  form  of  statistics.  Qualitative  data 
takes  on  increasing  importance  in  OT&E 
when  effectiveness  and  suitability  issues 
are  being  explored.  Many  techniques  are 
used  to  analyze  qualitative  data.  They  range 
from  converting  expressions  of  preference 
or  opinion  into  numerical  values  to  estab¬ 
lishing  a  consensus  by  committee.  For  ex¬ 
ample,  a  committee  may  assign  values  to 
parameters  such  as  "feel,"  "ease  of  use," 
"friendliness  to  the  user,"  and  "will  the  user 
want  to  use  it,"  on  a  scale  of  1-to-lO.  Care 
should  be  exercised  in  the  interpretation  of 
the  results  of  qualitative  evaluations.  For 
instance,  when  numbers  are  assigned  to  av¬ 
erage  evaluations  and  their  standard  devia¬ 
tions,  meanings  will  differ  from  quantitative 
data  averages  and  standard  deviations. 

13.7.1  Technical  Evaluation 

The  Services’  materiel  development  or¬ 
ganizations  are  usually  responsible  for 


oversight  of  all  aspects  of  DT&E  including 
the  technical  evaluation.  The  objectives  of  a 
technical  evaluation  are: 

•  To  assist  the  developers  by  providing 
information  relative  to  technical  perfor¬ 
mance;  qualification  of  components;  com¬ 
patibility,  interoperability,  vulnerability, 
lethality,  transportabihty,  reliability,  avail¬ 
ability  and  maintainability  (RAM);  man¬ 
power  and  personnel;  system  safety;  inte¬ 
grated  logistics  support;  correction  of  defi¬ 
ciencies;  accuracy  of  environmental  docu¬ 
mentation;  and  refinement  of  requirements; 

•  To  ensure  the  effectiveness  of  the  manu¬ 
facturing  process  of  equipment  and  proce¬ 
dures  through  production  qualification 
T&E; 

•  To  confirm  readiness  for  operational 
test  (OT)  by  ensuring  that  the  system  is 
stressed  beyond  the  levels  expected  in  the 
OT  environment; 

•  To  provide  information  to  the  decision 
authority  at  each  decision  point  regarding 
a  system's  technical  performance  and  readi¬ 
ness  to  proceed  to  the  next  phase  of  acqui¬ 
sition; 

•  To  determine  the  system's  operability 
in  the  required  climatic  and  realistic  battle¬ 
field  environments  to  include  natural,  in¬ 
duced,  and  countermeasure  environments 
(Reference  59). 

13.7.2  Operational  Evaluation 

The  independent  OT&E  authority  is  re¬ 
sponsible  for  the  operational  evaluation. 
The  objectives  of  an  operational  evaluation 
are: 

•  To  assist  the  developers  by  providing 
information  relative  to  operational  perfor¬ 
mance;  doctrine,  tactics,  training,  logistics; 
safety;  survivability;  manpower,  technical 
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publications;  RAM;  correction  of  deficien¬ 
cies;  accuracy  of  environmental  documen¬ 
tation;  and  refinement  of  requirements; 

•  To  assist  decision  makers  ensure  that 
only  systems  that  are  operationally  effec¬ 
tive  and  suitable  are  delivered  to  the  oper¬ 
ating  forces; 

•  To  provide  information  to  the  decision 
authority  at  each  decision  point  as  to  a 
system's  operational  effectiveness,  suitabil¬ 
ity  and  readiness  to  proceed  to  the  next 
phase  of  acquisition; 

•  To  assess,  from  the  user’s  viewpoint,  a 
system's  desirability,  considering  systems 
already  fielded,  and  the  benefits  or  burdens 
associated  with  the  system  (Reference  84). 


13.8  SUMMARY 

A  primary  consideration  in  identifying  in¬ 
formation  to  be  generated  by  an  evaluation 
program  is  having  a  clear  understanding  of 
the  decisions  the  information  will  support. 
The  importance  of  structuring  theT&E  pro¬ 
gram  to  support  the  resolution  of  critical 
issues  cannot  be  overemphasized.  It  is  the 
responsibihty  of  those  involved  in  the  evalu¬ 
ation  process  to  ensure  that  the  proper 
focus  is  maintained  on  key  issues,  the  T&E 
program  yields  information  on  critical  tech¬ 
nical  and  operational  issues,  all  data  sources 
necessary  for  a  thorough  evaluation  are 
tapped  and  evaluation  results  are  commu¬ 
nicated  in  an  effective  and  timely  manner. 
The  evaluation  process  should  be  evolu¬ 
tionary  throughout  the  acquisition  phases. 
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MODELING  AND  SIMULATION 
SUPPORT  TO  T&E 


14.1  INTRODUCTION 

This  chapter  discusses  the  applications  of 
modeling  and  simulation  in  test  and  evalu¬ 
ation  (T&E).  The  need  for  modeling  and 
simulation  has  long  been  recognized,  as 
evidenced  by  this  quotation  from  the  USAF 
Scientific  Advisory  Board  in  June  1965: 

Prediction  of  combat  effectiveness  can 
only  be,  and  therefore  must  be,  made 
by  using  the  test  data  in  analytical 
procedures.  This  analysis  usually  in¬ 
volves  some  type  of  model,  simula¬ 
tion,  or  game  (i.e.,  the  tools  of  opera¬ 
tions  or  research  analysis).  It  is  the 
exception  and  rarely,  that  the  'end  re¬ 
sult'  i.e.,  combat  effectiveness,  can  be 
deduced  directly  from  test  measure¬ 
ments. 

In  mandating  T&E  early  in  the  acquisition 
process  (i.e.,  before  Milestone  II),  Depart¬ 
ment  of  Defense  (DOD)  5000.2-R  encour¬ 
ages  the  use  of  modeling  and  simulation  as 
a  source  of  T&E  data.  For  instance,  the 
Armored  Family  of  Vehicles  program  used 
more  than  60  models,  simulations  and  other 
test  data  to  support  system  concept  explo¬ 
ration.  The  reliance  on  modeling  and  simu¬ 
lation  by  this  and  other  acquisition  pro¬ 
grams  provides  the  T&E  community  with 
valuable  information  which  can  increase 
confidence  levels,  decrease  field  test  time 
and  costs,  and  provide  data  for  pretest 
prediction  and  post-test  validation.  The 
Defense  Modeling  and  Simulation  Office 
(DMSO),  working  for  the  Director  Defense 


Research  and  Engineering,  is  developing 
Office  of  the  Secretary  of  Defense  ((DSD) 
guidance  on  the  application  of  modeling 
and  simulation  to  the  acquisition  process. 
The  DMSO  has  formed  the  Defense  Model¬ 
ing,  Simulation  and  Tactical  Technology 
Information  Analysis  Center  and  the  Mod¬ 
eling  and  Simulation  Operational  Support 
Activity  to  provide  assistance  to  program 
offices  and  the  acquisition  community  at 
large. 

This  chapter  discusses  using  modeling  and 
simulation  to  increase  the  efficiency  of  the 
T&E  process,  reduce  time  and  cost,  provide 
otherwise  unattainable  and  immeasurable 
data,  and  provide  more  timely  and  valid 
results. 

14.2  TYPES  OF  MODELS 
AND  SIMULATIONS 

The  term  "modeling  and  simulation"  is  of¬ 
ten  associated  with  huge  digital  computer 
simulations;  but  it  also  includes  manual 
and  man-in-the-loop  war  games,  test  beds, 
hybrid  laboratory  simulators  and  proto¬ 
types. 

A  mathematical  model  is  an  abstract  repre¬ 
sentation  of  a  system  that  provides  a  means 
of  developing  quantitative  performance  re¬ 
quirements  from  which  candidate  designs 
can  be  developed.  Static  models  are  those 
that  depict  conditions  of  state  while  dy¬ 
namic  models  depict  conditions  that  vary 


with  time,  such  as  the  action  of  an  autopilot 
in  controlling  an  aircraft.  Simple  dynamic 
models  can  be  solved  analytically,  and  the 
results  represented  graphically. 

According  to  a  former  Director,  Defense 
Test  and  Evaluation  (Reference  119),  simu¬ 
lations  used  in  T&E  can  be  divided  into 
three  categories: 

Constructive  Simulations:  Computer 
simulations  are  strictly  mathematical 
representations  of  systems  and  do  not 
employ  any  actual  hardware.  They 
may,  however,  incorporate  some  of 
the  actual  software  that  might  be  used 
in  a  system.  Early  in  a  system's  life 
cycle,  computer  simulations  can  be 
expected  to  provide  the  most  system 
evaluation  irvformation.  In  many  cases, 
computer  simulations  can  be  readily 
developed  as  modifications  of  existing 
simulations  for  similar  systems.  For 
example,  successive  generations  of 
AIM-7  missile  simulations  have  been 
effectively  used  in  test  and  evaluation. 

Virtual  Simulations:  A  system  test  bed 
usually  differs  from  a  computer  simu¬ 
lation  as  it  contains  some,  but  not  nec¬ 
essarily  all,  of  the  actual  hardware  that 
will  be  a  part  of  the  system.  Other 
elements  of  the  system  are  either  not 
incorporated  or,  if  they  are  incorpo¬ 
rated,  are  in  the  form  of  computer 
simulations.  The  system  operating  en¬ 
vironment  (including  threat)  may  ei¬ 
ther  be  physically  simulated,  as  in  the 
case  of  a  flying  test  bed,  or  computer 
simulated,  as  in  the  case  of  a  labora¬ 
tory  test  bed.  Aircraft  cockpit  simula¬ 
tors  used  to  evaluate  pilot  performance 
are  good  examples  of  system  test  beds. 
As  development  of  a  system 
progresses,  more  subsystems  become 
available  in  hardware  form.  These  sub¬ 
systems  can  be  incorporated  into  sys¬ 
tem  test  beds  that  typically  provide  a 


great  deal  of  the  system  evaluation 
information  used  during  the  middle 
part  of  a  system's  development  cycle. 

Anothertypeofvirtual  simulation  used 
in  T&E  is  the  system  prototype.  Unlike 
the  system  test  bed,  all  subsystems  are 
physically  incorporated  in  a  system 
prototype.  The  system  prototype  may 
closely  represent  the  final  system  con¬ 
figuration,  depending  on  the  state  of 
development  of  the  various  sub¬ 
systems  that  compose  it.  Preproduction 
prototype  missiles  and  aircraft  used  in 
operational  testing  by  the  Services  are 
examples  of  this  class  of  simulation. 
As  system  development  proceeds, 
eventually  all  subsystems  will  become 
available  for  incorporation  in  one  or 
more  system  prototypes.  Hardware- 
in-the-loop  (HWIL)  simulators  or  full- 
up  man-in-the-loop  system  simulators 
may  provide  the  foundation  for  con¬ 
tinuous  system  testing  and  improve¬ 
ment.  These  simulators  can  provide 
the  basis  for  transitioning  hardware 
and  software  into  operationally  realis¬ 
tic  training  devices  with  mission  re¬ 
hearsal  capabilities.  Operational  test¬ 
ing  of  these  prototypes  frequently  pro¬ 
vides  much  of  the  system  evaluation 
information  needed  for  a  decision  on 
full-scale  production  and  deployment. 

Live  Simulations:  Some  say  that  ev¬ 
erything  except  global  combat  is  a 
simulation,  even  limited  regional  en¬ 
gagements.  Live  exercises  where 
troops  use  equipment  under  actual 
environmental  conditions  approaches 
real  life  in  combat  while  conducting 
peacetime  operations.  Training  exer¬ 
cises  and  other  live  simulations  pro¬ 
vide  a  testing  ground  with  real  data 
on  actual  hardware,  software  and 
human  performance  when  subjected 
to  stressful  conditions.  These  data 
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Figure  1 4-1 .  The  Simulation  Spectrum 


can  be  used  to  validate  the  models 
and  simulations  used  in  an  acquisi¬ 
tion  program. 

As  illustrated  in  Figure  14-1,  there  is  a 
continuous  spectrum  of  simulation  types 
with  the  pure  computer  simulation  at  one 
end  and  the  pure  hardware  prototype  at 
the  other  end. 

14.3  VALIDITY  OF  MODELING 
AND  SIMULATION 

Simulations  are  not  a  substitute  for  live 
testing.  There  are  many  things  that  cannot 
be  adequately  simulated  by  computer  pro¬ 
grams;  among  them  are  the  process  of  deci¬ 
sion  and  the  proficiency  of  personnel  in  the 
performance  of  their  functions.  Therefore, 
models  and  simulations  are  not  a  total  sub¬ 
stitution  for  physical  tests  and  evaluations. 
Simulations,  manual  and  computer-de¬ 
signed,  can  complement  and  increase  the 
validity  of  live  tests  and  evaluations  by 
proper  selection  and  application.  Figure 
14-2  contrasts  the  test  criteria  that  are  con¬ 
ducive  to  modeling  and  simulation  versus 
physical  testing.  Careful  selection  of  the 
simulation,  knowledge  of  its  application 
and  operation  and  meticulous  selection  of 
input  data  will  produce  representative  and 
valid  results. 

The  important  element  in  using  a  simula¬ 
tion  is  to  select  one  that  is  representative 
and  either  addresses,  or  is  capable  of  being 
modified  to  address,  the  level  of  detail  (is¬ 
sues,  thresholds  and  objectives)  under  in¬ 
vestigation.  Models  and  simulations  must 
be  approved  for  use  through  verification, 
validation  and  accreditation  processes  DoD 
Directive  (DoDD)  5000.59).  Verification  is 
the  process  of  determining  that  a  model 
implementation  accurately  represents  the 
developer's  conceptual  description  and 
specifications.  Validation  is  the  process  of 
determining  (a)  the  manner  and  degree 


to  which  a  model  is  an  accurate  represen¬ 
tation  of  the  real-world  from  the  perspec¬ 
tive  of  the  intended  uses  of  the  model, 
and  (b)  the  confidence  that  should  be 
placed  on  this  assessment.  Accreditation 
is  the  official  certification  that  a  model  or 
simulation  is  acceptable  for  use  for  a  spe¬ 
cific  purpose. 

14.4  SUPPORT  TO  TEST  DESIGN 
AND  PLANNING 

14.4.1  Modeling  and  Simulation 
in  T&E  Planning 

Modeling  and  simulation  can  assist  in  the 
T&E  planning  process  and  can  reduce  the 
cost  of  testing.  In  Figure  14-3,  areas  of  par¬ 
ticular  application  include  scenario  devel¬ 
opment  and  the  timing  of  test  events;  the 
development  of  objectives,  essential  ele¬ 
ments  of  analysis,  and  measures  of  effec¬ 
tiveness;  the  identification  of  variables  for 
control  and  measurement;  and  the  devel¬ 
opment  of  data  collection,  instrumentation 
and  data  analysis  plans.  For  example,  us¬ 
ing  simulation,  the  test  designer  can  exam¬ 
ine  system  sensitivities  to  changes  in  vari¬ 
ables  to  determine  the  critical  variables  and 
their  ranges  of  values  to  be  tested.  The  test 
desiner  can  also  predict  the  effects  of  vari¬ 
ous  assumptions  and  constraints  and  evalu¬ 
ate  candidate  measures  of  effectiveness  to 
help  formulate  the  test  design. 

Caution  must  be  exercised  when  planning 
to  rely  on  simulations  to  obtain  test  data  as 
they  tend  to  be  expensive  to  develop  or 
modify,  difficult  to  integrate  with  data  from 
other  sources,  and  often  do  not  provide  the 
level  of  realism  required  for  operational 
tests.  Although  simulations  are  not  a  "cure- 
all,"  they  should  be  used  whenever  feasible 
as  another  source  of  data  for  the  evaluator 
to  consider  during  the  test  evaluation. 

Computer  simulations  may  be  used  to  test 
the  planning  for  an  exercise.  By  setting  up 
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Figure  14-2.  Values  of  Selected  Criteria  Conducive  to  Modeling  and  Simulation 
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Figure  14-3.  Modeling  and  Simulation  Application  in  Test  and  Evaluation 


and  running  the  test  exercise  in  a  simula¬ 
tion,  the  timing  and  scenario  may  be  tested 
and  validated.  Critical  events  may  include 
interaction  of  various  forces  that  test  the 
measures  of  effectiveness  and,  in  turn,  test 
objectives.  Further,  the  simulation  may  be 
used  to  verify  the  statistical  test  design  and 
the  instrumentation,  data  collection,  and 
data  analysis  plans.  Essentially,  the  pur¬ 
pose  of  computer  simulation  in  pretest 
planning  is  to  preview  the  test  to  evaluate 
ways  to  make  test  results  more  effective. 
Pretesting  attempts  to  optimize  test  results 
by  pointing  out  potential  trouble  spots.  It 
constitutes  a  test  setup  analysis,  which  can 
encompass  a  multitude  of  areas.  The  model- 
test-model  process  is  an  integrated  ap¬ 
proach  to  using  models  and  simulations  in 
support  of  pre-test  analysis  and  planning; 
conducting  the  actual  test  and  collecting 
data;  and  post-test  analysis  of  test  results 
along  with  further  validation  of  the  models 
using  the  test  data. 

As  an  example  of  simulations  used  in  test 
planning,  consider  a  model  that  portrays 
aircraft  versus  air  defenses.  The  model  can 
be  used  to  replicate  typical  scenarios  and 
provide  data  on  the  number  of  engage¬ 
ments,  air  defense  systems  involved,  air¬ 
craft  target,  length  and  quality  of  the  en¬ 
gagement,  and  a  rough  approximation  of 
the  success  of  the  mission  (i.e.,  if  the  aircraft 
made  it  to  the  target).  With  such  data  avail¬ 
able,  a  data  collection  plan  can  be  devel¬ 
oped  to  specify,  in  more  detail,  when  and 
where  data  should  be  collected,  from  which 
systems,  and  in  what  quantity.  The  results 
of  this  analysis  impact  heavily  on  long  lead- 
time  items  such  as  data  collection  devices 
and  data  processing  systems.  The  more 
specificity  available,  the  fewer  the  number 
of  surprises  that  will  occur  downstream. 
As  tactics  are  decided  upon  and  typical 
flight  paths  are  generated  for  the  scenario, 
an  analysis  can  be  prepared  on  the  flight 
paths  over  the  terrain  in  question;  and  a 


determination  can  be  made  regarding 
whether  the  existing  instrumentation  can 
track  the  numbers  of  aircraft  involved  in 
their  maneuvering  envelopes.  Alternative 
site  arrangements  can  be  examined  and 
trade-offs  can  be  made  between  the  amount 
of  equipment  to  be  purchased  and  the  types 
of  profiles  that  can  be  tracked  for  this 
particular  test.  Use  of  such  a  model  can 
also  highlight  numerous  choices  avail¬ 
able  to  the  threat  air  defense  system  in 
terms  of  opportunities  for  engagement 
and  practical  applications  of  doctrine  to 
the  specific  situations. 

14.4.2  Simulation,  Test  and  Evaluation 
Process  (STEP) 

In  STEP,  simulation  and  test  are  integrated, 
each  depending  on  the  other  to  be  effective 
and  efficient.  Simulations  provide  predic¬ 
tions  of  the  system's  performance  and  ef¬ 
fectiveness,  while  tests  are  part  of  a  strat¬ 
egy  to  provide  information  regarding  risk 
and  risk  mitigation,  to  provide  empirical 
data  to  validate  models  and  simulations, 
and  to  determine  whether  systems  are  op¬ 
erationally  effective,  suitable,  and  surviv- 
able  for  intended  use.  A  by-product  of  this 
process  is  a  set  of  models  and  simulations 
with  a  known  degree  of  credibility  provid¬ 
ing  the  potential  for  reuse  in  other  efforts 
(Figure  14-4). 

STEP  is  driven  by  mission  and  system  re¬ 
quirements.  The  product  of  STEP  is  infor¬ 
mation.  The  information  supports  acquisi¬ 
tion  program  decisions  regarding  techni¬ 
cal  risk,  performance,  system  maturity, 
operational  effect,  suitability,  and  surviv¬ 
ability.  STEP  applies  to  all  acquisition  pro¬ 
grams,  especially  Major  Defense  Acquisi¬ 
tion  Programs  (MDAPs)  and  Major  Auto¬ 
mated  Information  Systems  (MAIS). 

Throughout  STEP,  tests  are  conducted  to 
collect  data  for  evaluating  the  system  and 
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Figure  14-4.  STEP  Process 


refining  and  validating  models.  Through 
the  model-test-model  iteration  approach, 
the  sets  of  models  mature,  culminating  in 
accurate  representations  of  the  system  with 
appropriate  fidelity  which  can  be  used  to 
predict  system  performance  and  to  sup¬ 
port  the  acquisition  and  potentially  the 
training  communities. 

1 .  STEP  begins  with  the  Missions  Needs 
Statement  (N^S)  and  continues  thru  the 
life  cycle.  Top  level  requirements  are  used 
to  develop  alternative  concepts  and  select/ 
develop  digital  models  that  are  used  to 
evaluate  theater /campaign  and  mission/ 
battle-level  simulations.  Mission/battle 
level  models  are  used  to  evaluate  the  abil¬ 
ity  of  a  multiple  platform  force  package  to 
perform  a  specific  mission.  Mission  and 
functional  requirements  continue  to  be  re¬ 
fined,  and  the  system  reaches  the  prelimi¬ 
nary  design  stage. 

2.  Modeling  and  Simulation  (M&S)  is 
used  both  as  a  predictive  tool  and  with  test 
in  an  iterative  process  to  evaluate  the  sys¬ 
tem  design.  The  consequences  of  design 
changes  are  evaluated  and  help  translate 
the  most  promising  design  approach  into  a 
stable,  interoperable,  and  cost  effective  de¬ 
sign. 

3.  System  components  and  subsystems 
are  tested  in  a  laboratory  environment.  Data 
from  this  hardware  is  employed  in  the 
model-test-model  process.  Modeling  and 
Simulation  is  used  in  the  planning  of  tests 
to  support  a  more  efficient  use  of  resources. 
Simulated  tests  can  be  run  on  virtual  ranges 
to  conduct  rehearsals  and  determine  if  test 
limitations  can  be  resolved.  STEP  tools  are 
used  to  provide  data  for  determining  the 
real  component  or  subsystem's  perfor¬ 
mance  and  interaction  with  other  compo¬ 
nents.  Modeling  and  simulation  is  used 
during  both  developmental  testing  (DT) 
and  operational  testing  (OT)  to  increase  the 


amount  of  data  and  supplement  the  live 
test  events  that  are  needed  to  meet  test 
objectives. 

4.  Periodically  throughout  the  acquisi¬ 
tion  process  the  current  version  of  the  sys¬ 
tem  under  development  should  be  reexam¬ 
ined  in  a  synthetic  operational  context  to 
reassess  its  military  worth.  This  is  one  of 
the  significant  aspects  of  STEP,  understand¬ 
ing  the  answer  to  the  question;  What  differ¬ 
ence  does  this  change  make?  in  the  system's 
performance. 

5.  STEP  does  not  end  with  fielding  and 
deployment  of  a  system,  but  continues  to 
the  end  of  the  system's  life  cycle.  STEP 
results  in  a  thoroughly  tested  system  with 
performance  and  suitability  risks  identi¬ 
fied.  A  by-product  is  a  set  of  models  and 
simulations  with  a  known  degree  of  cred¬ 
ibility  with  the  potential  for  reuse  in  other 
efforts.  New  test  data  can  be  applied  to 
models  to  incorporate  any  system  enhance¬ 
ments  and  further  validate  its  models. 

14.5  SUPPORT  TO  TEST  EXECUTION 

Simulations  can  be  useful  in  test  execution 
and  dynamic  planning.  With  funds  and 
other  restrictions  limiting  the  number  of 
times  that  a  test  may  be  repeated  and  each 
test  conducted  over  several  days,  it  is  man¬ 
datory  that  the  test  director  exercises  close 
control  over  the  conduct  of  the  test  to  en¬ 
sure  the  specific  types  and  quantities  of 
data  needed  to  meet  the  test  objectives  are 
being  gathered  and  to  ensure  adequate 
safety.  The  test  director  must  be  able  to 
make  minor  modifications  to  the  test  plan 
and  scenario  to  force  achievement  of  these 
goals.  This  calls  for  a  dynamic  (quick-look) 
analysis  capability  and  a  dynamic  plan¬ 
ning  capability.  Simulations  may  contrib¬ 
ute  to  this  capability.  For  example,  using 
the  same  simulation(s)  as  used  in  pre-test 
planning,  the  tester  could  input  data  gath¬ 
ered  during  the  first  day  of  the  exercise  to 
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determine  the  adequacy  of  the  data  to  ful¬ 
fill  the  test  objectives.  Using  this  data,  the 
entire  test  could  be  simulated.  Projected 
inadequacies  could  be  isolated,  and  the  test 
plans  could  be  modified  to  minimize  the 
deficiencies. 

Simulations  may  also  be  used  to  support  test 
control  and  to  ensure  safety.  For  example, 
during  missile  test  firings  at  White  Sands 
Missile  Range  (W SMR),  aerodynamic  simu¬ 
lations  of  the  proposed  test  were  run  on  a 
computer  during  actual  firings  so  that  real¬ 
time  missile  position  data  could  be  com¬ 
pared  continuously  to  the  simulated  mis¬ 
sile  position  data.  If  any  significant  varia¬ 
tions  occurred  and  if  the  range  safety  of¬ 
ficer  was  too  slow  (both  types  of  position 
data  were  displayed  on  plotting  boards), 
the  computer  issued  a  destruct  command. 

Simulations  can  be  used  to  augment  tests 
by  simulating  nontestable  events  and  sce¬ 
narios  .  Although  operational  testing  should 
be  accomplished  in  as  realistic  an  opera¬ 
tional  environment  as  possible,  pragmati¬ 
cally  some  environments  are  impossible  to 
simulate  for  safety  or  other  reasons.  Some 
of  these  include  the  environment  of  a 
nuclear  battlefield,  to  include  the  effects  of 
nuclear  bursts  on  friendly  and  enemy  ele¬ 
ments.  Others  include  two-sided  live  fir¬ 
ings  and  adequate  representation  of  other 
forces  to  ascertain  compatibility  and 
interoperability  data.  Instrumentation,  data 
collection  and  data  reduction  of  large  com¬ 
bined  armed  forces  (e.g.,  brigade,  division 
and  larger-sized  forces)  become  extremely 
difficult  and  costly.  Simulations  are  not 
restricted  by  safety  factors  and  can  realisti¬ 
cally  replicate  many  environments  that  are 
otherwise  unachievable  in  an  operational 
test  and  evaluation  (OT&E)  —  nuclear  ef¬ 
fects,  large  combined  forces,  electronic 
countermeasures  (ECM),  electronic 
counter-countermeasures  (ECCM)  and 
many  engagements. 


Usually,  insufficient  units  are  available  to 
simulate  the  organizational  relationships 
and  interaction  of  the  equipment  with  its 
operational  environment,  particularly  dur¬ 
ing  the  early  OT&E  conducted  using  proto¬ 
type  or  pilot  production-type  equipment. 
Simulations  are  not  constrained  by  these 
limitations.  Data  obtained  from  a  limited 
test  can  be  plugged  into  a  simulation  that  is 
capable  of  handling  many  of  the  types  of 
equipment  being  tested.  It  can  interface 
them  with  other  elements  of  the  blue  forces 
and  operate  them  against  large  elements  of 
the  red  forces  to  obtain  interactions. 

End-item  simulators  can  be  used  to  evalu¬ 
ate  design  characteristics  of  equipment  and 
can  be  used  to  augment  the  results  ob¬ 
tained  using  prototype  or  pilot  produc¬ 
tion-type  equipment  that  is  representative 
of  the  final  item.  The  simulator  may  be 
used  to  expand  test  data  to  obtain  the  re¬ 
quired  iterations  or  to  indicate  that  the 
human  interface  with  the  prototype  equip¬ 
ment  will  not  satisfy  the  design  require¬ 
ments. 

It  is  often  necessary  to  use  substitute  or 
surrogate  equipment  in  testing;  e.g.,  Ameri¬ 
can  equipment  is  used  to  represent  threat- 
force  equipment.  In  some  cases  the  substi¬ 
tute  equipment  may  have  greater  capabili¬ 
ties  than  the  real  equipment,  in  other  cases 
it  may  have  less.  Simulations  are  capable  of 
representing  the  real  characteristics  of 
equipment  and,  therefore,  can  be  used  as  a 
means  of  modifying  raw  data  collected 
during  the  test  to  reflect  real  characteris¬ 
tics. 

As  an  example,  if  the  substitute  equipment 
is  an  AAA  gun  with  a  tracking  rate  of  30 
degrees  per  second  and  the  equipment  for 
which  it  is  substituted  has  a  tracking  rate  of 
45  degrees  per  second,  the  computer  simu¬ 
lation  could  be  used  to  augment  the  col- 
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lected,  measured  data  by  determining  how 
many  rounds  could  have  been  fired  against 
each  target  or  whether  targets  that  were 
missed  because  the  tracking  rate  was  too 
slow  could  have  been  engaged  by  the  ac¬ 
tual  equipment.  Consideration  of  other  dif¬ 
fering  factors  simultaneously  could  have  a 
plus  or  minus  synergistic  effect  on  test  re¬ 
sults. 

14.6  SUPPORT  TO  ANALYSIS 
AND  TEST  REPORTING 

Modeling  and  simulation  may  be  used  in 
post-test  analysis  to  extend  and  generalize 
results  and  to  extrapolate  to  other  condi¬ 
tions.  The  difficulty  of  instrumenting  and 
controlling  large  exercises  and  collecting 
and  reducing  the  data  and  resource  costs, 
to  some  degree,  limits  the  size  of  T&E.  This 
makes  the  process  of  determining  the  suit¬ 
ability  of  equipment  to  include  compatibil¬ 
ity,  interoperability,  organization,  etc.,  a 
difficult  one.  To  a  large  degree  the  limited 
interactions,  interrelationships  and  com¬ 
patibility  of  large  forces  may  be  supple¬ 
mented  by  using  actual  data  collected  during 
the  test  and  applying  it  in  the  simulatioa 

Simulations  can  be  used  to  extend  test 
results,  save  considerable  energy  (fuel 


and  manpower),  and  save  money  by  re¬ 
ducing  the  need  to  repeat  data  points  to 
improve  the  statistical  sample  or  to  deter¬ 
mine  overlooked  or  directly  unmeasured 
parameters.  Sensitivity  analyses  can  be 
run  using  simulations  to  evaluate  the  ro¬ 
bustness  of  the  design. 

In  analyzing  the  test  results,  data  can  be 
compared  to  the  results  predicted  by  the 
simulations  used  early  in  the  planning  pro¬ 
cess.  Thus,  the  simulation  is  validated  by 
the  actual  live  test  results,  but  the  test  re¬ 
sults  are  also  validated  by  the  simulation. 

14.7  SUMMARY 

Modeling  and  simulation  in  T&E  can  be 
used  for  concept  evaluation,  extrapolation, 
isolation  of  design  effects,  efficiency,  repre¬ 
sentation  of  complex  environments,  and 
overcoming  inherent  limitations  in  actual 
testing.  The  use  of  modeling  and  simula¬ 
tion  can  validate  test  results,  increase  con¬ 
fidence  levels,  reduce  test  costs  and  pro¬ 
vide  opportunities  to  shorten  the  overall 
acquisition  cycle  by  providing  more  data 
earlier  for  the  decision-maker.  But  it  does 
take  time  and  funding  to  bring  models  and 
simulations  along  to  the  point  that  they  are 
useful  during  an  acquisition. 
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TEST  RESOURCE 


15.1  INTRODUCTION 

This  chapter  describes  the  various  types  of 
resources  available  for  testing,  explains  test 
resource  planning  in  the  Services,  and  dis¬ 
cusses  the  ways  in  which  test  resources  are 
funded. 

According  to  Department  of  Defense  (DoD) 
5000.2-R,  the  term  "test  resources"  is  a  col¬ 
lective  term  that  encompasses  elements 
necessary  to  plan,  conduct,  collect  and  ana¬ 
lyze  data  from  a  test  event  or  program. 
These  elements  include:  funding  (to  de¬ 
velop  new  resources  or  use  existing  ones), 
manpower  for  test  conduct  and  support, 
test  articles,  models,  simulations,  threat 
simulators,  surrogates,  replicas,  test-beds, 
special  instrumentation,  test  sites,  targets, 
tracking  and  data  acquisition  instrumenta¬ 
tion,  equipment  (for  data  reduction,  com¬ 
munications,  meteorology,  utilities,  pho¬ 
tography,  calibration,  security,  recovery, 
maintenance  and  repair),  frequency  man¬ 
agement  and  control,  and  base  /  facility  sup¬ 
port  services.  "Testing  shall  be  planned  and 
conducted  to  take  full  advantage  of  exist¬ 
ing  investment  in  DoD  ranges,  facilities, 
and  other  resources,  whenever  practical, 
unless  otherwise  justified  in  the  Test  and 
Evaluation  Master  Plan,"  (DoD  5000.2-R). 

Key  DoD  test  resources  are  in  great  de¬ 
mand  by  competing  acquisition  programs. 
Often  special,  unique  or  one-of-a-kind  test 
resources  must  be  developed  specifically 
for  the  test  program.  It  is  imperative  that 
the  requirements  for  these  test  resources  be 
identified  early  in  the  acquisition  process 


so  adequate  funding  can  be  allotted  for 
their  development,  and  they  will  be  avail¬ 
able  when  the  test  is  scheduled. 

15.2  OBTAINING  TEST  RESOURCES 

15.2.1  Identify  Test  Resources 
and  Instrumentation 

As  early  as  possible,  but  not  later  than  the 
start  of  the  Engineering  and  Manufactur¬ 
ing  Development  Phase,  the  test  facilities 
and  instrumentation  requirements  to  con¬ 
duct  operational  tests  should  be  identified 
and  a  tentative  schedule  of  test  activities 
prepared.  This  information  is  recorded  in 
the  T est  and  Evaluation  Master  Plan  (TEMP) 
and  Service  test  resource  documentation. 

15.2.2  Require  Multi-Service  OT&E 

Multi-Service  operational  test  and  evalua¬ 
tion  (OT&E)  should  be  considered  for 
weapon  systems  requiring  new  operational 
concepts  involving  other  Services.  If  multi- 
Service  testing  is  used,  an  analysis  of  the 
impact  of  demonstration  on  time  and  re¬ 
sources  needed  to  execute  the  multi-Ser- 
vice  tests  should  be  conducted  before  the 
Milestone  II  decision. 

15.2.3  Military  Construction  Program 
Facilities 

Some  programs  cannot  be  tested  without 
Military  Construction  Program  facilities. 
To  construct  these  facilities  will  require 


long  lead  times;  therefore,  early  planning 
must  be  done  to  ensure  that  the  facilities 
will  be  ready  when  required. 

15.2.4  Test  Sample  Size 

The  primary  basis  for  the  test-sample  size  is 
usually  based  on  one  or  more  of  the  follow¬ 
ing: 

•  Analysis  of  test  objectives; 

•  Statistical  significance  of  test  results  at 
some  specified  confidence  level; 

•  Availability  of  test  vehicles,  items,  etc.; 

•  Support  resources  or  facilities  avail¬ 
able; 

•  Time  available  for  the  test  program. 

15.2.5  Test  Tennination 

One  should  not  hesitate  to  terminate  a  test 
before  its  completion  if  it  becomes  clear 
that  the  main  objective  of  the  test  is 
unachievable  (due  to  hardware  failure,  un¬ 
availability  of  resources,  etc.)  or  if  addi¬ 
tional  samples  will  not  change  the  outcome 
and  conclusions  of  the  test. 

15.2.6  Budget  for  Test 

The  Acquisition  Strategy,  TEMP  and  bud¬ 
geting  documents  should  be  reviewed  regu¬ 
larly  to  ensure  that  there  are  adequate  iden¬ 
tified  testing  funds  relative  to  development 
and  fabrication  funds. 

The  Acquisition  Strategy,  TEMP  and  bud¬ 
geting  documents  need  careful  scrutiny  to 
ensure  that  there  are  adequate  contingency 
funds  to  cover  correction  of  difficulties  at  a 
level  that  matches  industry /government 
experience  on  the  contract.  (Testing  to 
correct  deficiencies  found  during  testing. 


without  sufficient  funding  for  proper  cor¬ 
rection,  results  in  Band-Aid  approaches, 
which  require  corrections  at  a  later  and 
more-expensive  time  period.) 

15.2.7  Test  Articles 

A  summary  of  important  test  planning 
items  that  were  identified  by  the  Defense 
Science  Board  (DSB)  is  provided  below: 

•  Ensure  that  the  whole  system,  includ¬ 
ing  the  system  user  personnel,  is  tested. 
Realistically  test  the  complete  system,  in¬ 
cluding  hardware,  software,  people  and  all 
interfaces.  Get  user  involved  from  the  start 
and  understand  user  limitations; 

•  Ascertain  that  sufficient  time  and  test 
articles  are  planned.  When  the  technology 
is  stressed,  the  higher  risks  require  more 
test  articles  and  time; 

•  In  general,  parts,  subsystems  and  sys¬ 
tems  should  be  proven  in  that  order  before 
incorporating  them  into  the  next  higher 
assembly  for  more  complete  tests.  The  in¬ 
strumentation  should  be  planned  to  permit 
diagnosis  of  trouble; 

•  Major  tests  should  never  be  repeated 
without  an  analysis  of  failure  and  correc¬ 
tive  action.  Allow  for  delays  of  this  nature. 

15.2.8  Major  Range 
and  Test  Facility  Base 

All  Services  operate  ranges  and  test  facili¬ 
ties  for  test,  evaluation  and  training  pur¬ 
poses.  Twenty-one  of  these  activities  con¬ 
stitute  the  DoD  Major  Range  and  Test  Facil¬ 
ity  Base  (MRTFB,  DoD  Directive  (DoDD) 
3200.11).  This  MRTFB  is  described  as  "a 
national  asset  which  shall  be  sized,  oper¬ 
ated,  and  maintained  primarily  for  DoD 
test  and  evaluation  support  missions,  but 
also  is  available  to  all  users  having  a  valid 
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requirement  for  its  capabilities.  The  MRTFB 
consists  of  a  broad  base  of  T&E  [test  and 
evaluation]  activities  managed  and  oper¬ 
ated  under  uniform  guidelines  to  provide 
T&E  support  to  DoD  Components  respon¬ 
sible  for  developing  or  operating  materiel 
and  weapon  systems,"  (Reference  21  A).  The 
list  of  MRTFB  activities  and  their  locations 
are  shown  on  Figure  15-1.  Summaries  of 
the  capabilities  of  each  of  these  activities 
(with  points  of  contact  listed  for  further  infor¬ 
mation)  may  be  found  in  DoD  3200.11-D. 

The  MRTFB  facilities  are  available  for  use 
by  all  the  Services,  other  U.S.  government 
agencies  and,  in  certain  cases,  allied  foreign 
governments  and  contractor  organizations. 
Scheduling  is  based  on  a  priority  system; 
and  costs  for  usage  are  billed  uniformly,  as 
stated  in  DoDD  3200.1 1 .  The  Deputy  Direc¬ 
tor,  Resources  and  Investments  (DTSE&E), 
sets  policy  for  the  composition,  use  and  test 
program  assignments  of  the  MRTFB.  In 
turn,  the  individual  Services  must  fund, 
manage  and  operate  their  activities.  They 
are  reimbursed  for  direct  costs  by  each  user 
of  the  activity.  The  Joint  Program  Office, 
T&E,  the  operating  arm  of  the  T&E  Board 
of  Operating  Directors,  has  sponsored  the 
development  of  a  Joint  Test  Assets  Data¬ 
base  which  lists  MRTFB  and  Operational 
Test  Agency  (OTA)  test  facilities,  test  area 
and  range  data,  instrumentation  and  test 
systems.  This  database  can  be  accessed  via 
the  TECNET/TECWeb. 

The  DoD  components  wishing  to  use  an 
MRTFB  activity  must  provide  timely  and 
complete  notification  of  their  requirements, 
such  as  special  instrumentation  or  ground- 
support  equipment  requirements,  to  the 
particular  activity  using  the  documenta¬ 
tion  formats  prescribed  by  Document  501- 
84,  Universal  Documentation  System  Hand¬ 
book,  issued  by  the  Range  Commanders 
Council.  The  requirements  must  be  stated 
in  the  TEMP  discussed  below.  Personnel  at 


the  MRTFB  activity  will  coordinate  with 
and  assist  prospective  users  with  their  T&E 
planning,  to  include  conducting  trade-off 
analyses  and  test  scenario  optimization 
based  on  test  objectives  and  test  support 
capabilities. 

15.2.9  Project  Reliance 

In  response  to  a  stated  need  to  consolidate 
DoD  activities  (Defense  Management  Re¬ 
view  Directive  922),  the  Office  of  the  Secre¬ 
tary  of  Defense  (OSD)  T&E  organizations 
have  initiated  a  process  to  review  and  cen¬ 
tralize  various  types  of  system  testing  in¬ 
frastructures  at  designated  Service  test  fa¬ 
cilities.  Project  Reliance  is  focused  on  more 
economical  operations,  allocating  scarce 
funds  for  modernization  and  eliminating 
unwarranted  duplication.  The  T&E  Reli¬ 
ance  and  Investment  Board,  under  the  T &E 
Board  of  Operating  Directors  ((Test  and 
Evaluation  Command  (TECOM),  Naval  Air 
Warfare  Center  (NAWC),  and  the  Air  Force 
Materiel  Command  (AFMC),  provides  tech¬ 
nical  leadership,  vision,  oversight,  and  re¬ 
view  for  all  Service  T&E  investment  plan¬ 
ning  activities  to  foster  development  of 
joint  investment  initiatives,  to  ensure  the 
development  and  sustainment  of  an  effec¬ 
tive  and  efficient  defense  T&E  capability, 
to  prevent  unwarranted  duplication  of  DoD 
T&E  capabilities,  and  to  optimize  the  Ser¬ 
vices'  investments  in  T&E  capabilities.  As  a 
follow  on  to  the  Reliance  process.  Congress 
directed  a  study  (Vision  21)  of  the  potential 
for  consolidation  of  laboratory  and  testing 
capabilities  to  further  reduce  the  incidence 
of  duplicative  efforts. 

15.2.10  Test  and  Evaluation 
Resources  Committee  (TERC) 

In  1994  the  Principal  Deputy  Under  Sec¬ 
retary  of  Defense  for  Acquisition  and 
Technology  (USD(A&T))  directed  that  the 
Director,  Text,  Systems  Engineering,  and 
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Figure  15-1.  DoD  Major  Range  and  Test  Facility 


Evaluation  establish  and  chair  a  steering 
group  that  would  oversee  the  acquisition 
and  integration  of  aU  training  and  associ¬ 
ated  test  range  instrumentation  and  de¬ 
velop  related  policy.  The  Etefense  Test  and 
Training  Steering  Group  subsequently  char¬ 
tered  the  TERC  to  manage  the  implementa¬ 
tion  of  the  Joint  Training  and  Test  Range 
Roadmap  and  execute  the  Central  Test  and 
Evaluation  Investment  Program  (CTEIP). 
TheCTEIP  provides  a  mechanism  for  the 
development  and  acquisition  of  new  test 
capabilities  to  satisfy  multi-Service  testing 
requirements. 

15.2.11  Service  Test 
Facilities 

There  are  other  test  resources  available  be¬ 
sides  MRTFB.  The  tester  can  determine 
resources  available  by  contacting  his/her 
Service  headquarters  staff  element  or  if 
within  the  Army,  by  consulting  documents 
such  as  the  Army  TECOM  Test  Facilities 
Register,  the  Operational  Test  and  Evalua¬ 
tion  Command  (OPTEC)  Operational  Test 
Instrumentation  Guide  and  other  Army 
test  agency  and  range  documents.  Infor¬ 
mation  on  specific  Navy  test  resources  is 
found  in  user  manuals  published  by  each 
range  and  the  Commander  Operational  Test 
and  Evaluation  Force  (COMOPTEVFOR) 
catalog  of  available  support. 

15.3  TEST  RESOURCE  PLANNING 

The  development  of  special  test  resources 
to  support  a  weapon  system  test  can  be 
costly  and  time-consuming.  This,  coupled 
with  the  competition  for  existing  test  re¬ 
sources  and  facilities,  requires  that  early 
planning  be  accomplished  to  determine  all 
test  resource  requirements  for  weapon  sys¬ 
tem  T&E.  The  tester  must  use  government 
facilities  whenever  possible  instead  of  fund¬ 
ing  construction  of  contractor  test  capabili¬ 
ties. 


Problems  associated  with  range  and  facil¬ 
ity  planning  are  that  major  systems  tend  to 
get  top  priority;  i.e.,  B-IB,  M-1,  etc.  Range 
schedules  are  often  in  conflict  due  to  sys¬ 
tem  problems,  which  cause  schedule  de¬ 
lays  during  testing;  and  there  is  often  a 
shortage  of  funds  to  complete  testing. 

15.3.1.5.1  TEMP  Resource  Requirements 

The  program  manager  (PM)  must  state  all 
key  test  resource  requirements  in  the  TEMP 
and  must  include  items  such  as  unique 
instrumentation,  threat  simulators,  surro¬ 
gates,  targets  and  test  articles.  Included  in 
the  TEMP  are  a  critical  analysis  of  antici¬ 
pated  resource  shortfalls,  their  effect  on 
system  T&E  and  plans  to  correct  resource 
deficiencies .  As  the  preliminary  TEMP  must 
be  prepared  for  Milestone  I,  initial  test  re¬ 
source  planning  must  be  accomplished 
during  the  Concept  Exploration  Phase. 
Refinements  and  reassessments  of  test  re¬ 
source  requirements  are  included  in  each 
TEMP  update.  The  guidance  for  the  con¬ 
tent  of  the  test  resource  summary  (Part  V) 
of  the  TEMP  is  in  Appendix  III  -  Test  and 
Evaluation  Master  Plan,  DoD  5000.2-R 
(Table  15-1).  Once  identified,  the  PM  must 
then  work  within  the  Service  headquarters 
and  range  management  structure  to  assure 
the  assets  are  available  when  needed. 

15.3.2  Service  Test  Resource  Planning 

More-detailed  listings  of  required  test  re¬ 
sources  are  generated  in  conjunction  with 
the  detailed  test  plans  written  by  the  mate¬ 
riel  developer  and  operational  tester.  These 
test  plans  describe  test  objectives,  measures 
of  effectiveness  (MOEs),  scenarios  and  spe¬ 
cific  test  resource  requirements. 

15.3.2.1  Army  Test  Resource  Planning 

In  the  Army,  the  tester  prepares  input  to 
the  TEMP  and  the  Test  and  Evaluation  Plan 
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Table  15-1.  TEMP  Test  Resource  Summary  Section 

PART  V— TEST  AND  EVALUATION  RESOURCE  SUMMARY 

Provide  a  summary  (preferably  in  a  table  or  matrix  format)  of  all  key  test  and  evalua¬ 
tion  resources,  both  government  and  contractor,  that  will  be  used  during  the  course 
of  the  acquisition  program. 

The  TEMP  should  project  the  key  resources  necessary  to  accomplish  demonstration 
and  validation  testing  and  early  operational  assessment.  The  TEMP  should  estimate, 
to  the  degree  known  at  Milestone  I,  the  key  resources  necessary  to  accomplish  de¬ 
velopmental  test  and  evaluation,  live  fire  test  and  evaluation,  and  operational  test  and 
evaluation.  These  should  include  elements  of  the  National  Test  Facilities  Base  (which 
incorporates  the  Major  Range  and  Test  Facility  Base  (MRTFB),  capabilities  desig¬ 
nated  by  industry  and  academia,  and  MRTFB  test  equipment  and  facilities),  unique 
instrumentation,  threat  simulators,  and  targets.  As  system  acquisition  progresses, 
the  preliminary  test  resource  requirements  shall  be  reassessed  and  refined  and  sub¬ 
sequent  TEMP  updates  shall  reflect  any  changed  system  concepts,  resource  re¬ 
quirements,  or  updated  threat  assessment.  Any  resource  shortfalls  which  introduce 
significant  test  limitations  should  be  discussed  with  planned  corrective  action  out¬ 
lined.  Specifically,  identify  the  following  test  resources: 

TEST  ARTICLES 

•  TEST  SITES  AND  INSTRUMENTATION 

•  TEST  SUPPORT  EQUIPMENT 

•  THREAT  SYSTEMS/SIMULATORS 
TEST  TARGETS  AND  EXPENDABLES 
OPERATIONAL  FORCE  TEST  SUPPORT 

•  SIMULATORS,  MODELS  AND  TEST-BEDS 
SPECIAL  REQUIREMENTS 

•  TEST  AND  EVALUATION  FUNDING  REQUIREMENTS 

•  MANPQWER/PERSONNEL  TRAINING 

SOURCE:  DOD5000.2.R 
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(TEP),  the  primary  planning  documents 
for  developmental  and  OT &E  of  the  weapon 
system.  These  documents  should  be  pre¬ 
pared  early  in  the  acquisition  cycle  (at  the 
beginning  of  the  Concept  Demonstration 
Phase).  They  describe  the  entire  T&E  strat¬ 
egy  including  critical  issues,  test  method¬ 
ology,  MOEs  and  all  necessary  test  re¬ 
sources.  The  TEMP  and  TEP  provide  the 
primary  input  to  the  Outline  Test  Plan 
(OTP),  which  contains  a  detailed  descrip¬ 
tion  of  each  identified  required  test  re¬ 
source,  where  and  when  it  is  to  be  pro¬ 
vided,  and  the  providing  organization. 

The  tester  must  coordinate  the  OTP  with  all 
major  commands  or  agencies  expected  to 
provide  test  resources.  Then,  the  OTP  is 
submitted  to  the  Resource  Management 
Division,  HQ,  OPTEC,  for  review  by  the 
Test  Schedule  and  Review  Committee 
(TSARC)  and  for  incorporation  into  the 
Army's  Five-Year  Test  Program  (FYTP). 
The  initial  OTP  for  each  test  should  be 
submitted  to  the  TSARC  as  soon  as  testing 
is  identified  in  the  TEMP.  Revised  OTPs 
are  submitted  as  more  information  becomes 
available  or  requirements  change,  but  a 
final  comprehensive  version  of  the  OTP 
should  be  submitted  at  least  18  months 
before  the  resources  are  required. 

The  TSARC  is  responsible  for  providing 
high-level,  centralized  management  of  T&E 
resource  planning.  The  TSARC  is  chaired 
by  the  Commanding  General  OPTEC  and 
consists  of  a  general  officer  or  equivalent 
representatives  from  the  Army  staff  and 
major  commands.  The  TSARC  meets  semi¬ 
annually  to  review  all  OTPs,  resolve  con¬ 
flicts  and  coordinate  all  identified  test  re¬ 
source  requirements  for  inclusion  in  the 
FYTP.  The  FYTP  is  a  formal  resource  task¬ 
ing  document  for  current  and  near-term 
tests  and  a  planning  document  for  tests 
scheduled  for  the  out-years.  All  OTPs  are 
reviewed  during  the  semiannual  reviews 


to  ensure  that  any  refinements  or  revisions 
are  approved  by  the  TSARC  and  reflected 
in  the  FYTP.  The  FYTP  is  produced  as  a 
hard-copy  by  OPTEC. 

The  TSARC-approved  OTP  is  a  tasking 
document  by  which  the  tester  requests 
Army  test  resources.  The  TSARC  coordi¬ 
nates  resource  requests,  sets  priorities,  re¬ 
solves  conflicts  and  schedules  resources. 
The  resultant  FYTP,  when  approved  by  the 
Deputy  Chief  of  Staff  for  Operations  and 
Plans  (DCSOPS),  HQ  DA,  is  a  formal  task¬ 
ing  document  that  reflects  the  agreements 
made  by  the  resource  providers  (Army 
Materiel  Command  (AMC),  Training  and 
Doctrine  Command  (TRAEXXl),  Forces 
Command  (FORSCOM),  etc.)  to  make  the 
required  test  resources  available  to  the  des¬ 
ignated  tests.  If  test  resources  from  another 
Service,  a  non-DoD  governmental  agency 
(such  as  the  Department  of  Energy  (DOE) 
or  NASA)  or  a  contractor  are  required,  the 
request  is  coordinated  by  the  OPTEC  Re¬ 
source  Management  Division.  For  example, 
the  request  for  a  range  must  be  made  at 
least  two  years  in  advance  to  ensure  avail¬ 
ability.  However,  due  to  the  long  lead  time 
required  to  schedule  these  non- Army  re¬ 
sources,  their  availability  cannot  be  guar¬ 
anteed  if  testing  is  delayed  or  retesting  is 
required.  The  use  of  resources  outside 
the  U.S.,  such  as  in  Canada,  Germany  or 
other  NATO  countries,  is  also  handled  by 
OPTEC. 

15.3.2.2  Navy  Test  Resource 
Planning 

In  the  Navy,  the  developing  agency  and  the 
operational  tester  are  responsible  for  iden¬ 
tifying  the  specific  test  resources  required 
in  testing  the  weapon  system.  In  develop¬ 
ing  requirements  for  test  resources,  the  PM 
and  operational  test  director  (OTD)  refer 
to  documents  such  as  the  Mission  Need 
Statement  (MNS),  Acquisition  Strategy, 
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Navy  Decision  Coordinating  Paper 
(NDCP),  Operational  Requirement  Docu¬ 
ment  (ORD),  threat  assessments, 
SECNAV  Instr  5000.2B,  and  the  OTD 
Guide  (Commander,  Operation  Test  and 
Evaluation  Force  (Navy)  (COM- 
OPTEVFOR)  Instruction  3960.1D).  Upon 
Chief  of  Naval  Operations  (CNO)  ap¬ 
proval,  the  TEMP  becomes  the  control¬ 
ling  management  document  for  all  T&E 
of  the  weapon  system.  It  constitutes  di¬ 
rection  by  the  CNO  to  conduct  the  T&E 
program  defined  in  the  TEMP,  including 
the  commitment  of  research,  develop¬ 
ment,  test  and  evaluation  (RDT&E)  fi¬ 
nancial  support  and  of  fleet  units  and 
schedules.  It  is  prepared  by  the  PM,  who 
is  provided  OT&E  input  by  the 
COMOPTE  VFOR  Operational  Test  Direc¬ 
tor.  The  TEMP  defines  all  T&E  (DT&E, 
OT&E  and  production  acceptance  test  and 
evaluation  (PAT&E))  to  be  conducted  for 
the  system  and  describes,  in  as  much 
detail  as  possible,  the  test  resources  re¬ 
quired. 

The  Navy  uses  its  operational  naval  forces 
to  provide  reaUstic  T&E  of  new  weapon 
systems.  Each  year,  the  CNO  (N-091)  com¬ 
piles  all  Fleet  support  requirements  for 
RDT&E  program  support  from  the  TEMPs 
and  publishes  the  CNO  Long-Range 
RDT&E  Support  Requirements  document 
for  the  budget  and  out-years.  In  addition,  a 
quarterly  forecast  of  support  requirements 
is  published  approximately  five  months 
before  the  Fleet  Employment  Scheduling 
Conference  for  the  quarter  in  which  the 
support  is  required.  These  documents  sum¬ 
marize  OT&E  requirements  for  Fleet  ser¬ 
vices  and  are  used  by  the  Fleet  for  schedul¬ 
ing  services  and  out-year  budget  projec¬ 
tions. 

Requests  for  use  of  range  assets  are  usually 
initiated  informally  with  a  phone  call  from 
the  PM  and/or  OTD  to  the  range  manager 


and  followed  by  formal  documentation. 
Requests  for  Fleet  support  are  usually  more 
formal.  The  COMOPTEVFOR,  in  coordi¬ 
nation  with  the  PM,  forwards  the  TEMP 
and  a  Fleet  RDT&E  Support  Request  to  the 
CNO.  Upon  approval  of  the  request,  the 
CNO  tasks  the  Fleet  Commander  in  Chief 
(CINC)  by  letter  or  message  to  coordinate 
with  OPTEVFOR  to  provide  the  requested 
support. 

Use  of  most  Navy  ranges  must  be  sched¬ 
uled  at  least  a  year  in  advance.  Each  range 
consolidates  and  prioritizes  user  requests, 
negotiates  conflicts  and  attempts  to  sched¬ 
ule  range  services  to  satisfy  all  requests.  If 
the  desired  range  services  cannot  be  made 
available  when  required,  the  test  must  wait; 
or  the  CNO  resolves  the  conflict.  Because 
ranges  are  fully  scheduled  in  advance,  it  is 
difficult  to  accommodate  a  test  that  is  de¬ 
layed  or  requires  additional  range  time 
beyond  that  originally  scheduled.  Again, 
the  CNO  can  examine  the  effects  of  delays 
or  retest  requirements  and  issue  revised 
priorities,  as  required. 

Requests  for  use  of  non-Navy  OT&E  re¬ 
sources  are  initiated  by  COMOPTEVFOR. 
The  Operational  Test  and  Evaluation 
Force  (OPTEVFOR)  is  authorized  direct 
liaison  with  other  Service-independent 
OTAs  to  obtain  OTA-controlled  re¬ 
sources.  Requests  for  other  government- 
owned  resources  are  forwarded  to  the 
CNO  (N-091)  for  formal  submission  to 
the  Service  Chief  (for  Service  assets)  or  to 
the  appropriate  government  agency  (e.g., 
DOE  or  NASA).  Use  of  contractor  re¬ 
sources  is  usually  handled  by  the  PM, 
although  contractor  assets  are  seldom 
required  in  OT&E,  since  the  Fleet  is  used 
to  provide  an  operational  environment. 
Requests  for  use  of  foreign  ranges  are 
handled  by  the  N-091  Assistant  for  In¬ 
ternational  Research  and  Development 
(R&D). 
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15.3.2.3  Air  Force  Test 
Resource  Planning 

The  test  resources  required  for  T&E  of  an 
Air  Force  weapon  system  are  identified  in 
detail  in  the  Test  Resources  Plan  (TRP), 
which  is  prepared  by  the  responsible  Air 
Force  T&E  organization.  In  general,  the  Air 
Force  Operational  Tests  and  Evaluation 
Center  (AFOTEC)  is  the  test  organization 
for  OT&E  programs;  it  obtains  support  from 
a  Service  major  command  test  agency  for 
nonmajor  programs,  with  AFOTEC  direct¬ 
ing  and  providing  assistance,  as  required. 

During  the  Advanced  Planning  Phase  of  a 
weapon  system  acquisition  (five  to  six  years 
before  OT &E),  AFOTEC  prepares  the  OT &E 
section  of  the  first  full  T^,  coordinates  the 
TRP  with  all  supporting  organizations  and 
assists  the  resource  manager  (RM)  in  pro¬ 
gramming  required  resources.  The  resource 
requirements  listed  in  the  Resource  Infor¬ 
mation  Network  TRP  are  developed  by  the 
test  manager,  RM  and  test  support  group, 
using  sources  such  as  the  ORD  and  threat 
assessments.  The  TRP  should  specify,  in 
detail,  all  the  resources  necessary  to  suc¬ 
cessfully  conduct  a  test  when  it  is  entered  in 
the  Test  Resource  Information  Management 
System  (TRIMS). 

The  TRP  is  the  formal  means  by  which  test 
resource  requirements  are  communicated 
to  the  Air  Staff  and  to  the  appropriate  com¬ 
mands  and  agencies  tasked  to  supply  the 
needed  resources.  Hence,  if  a  required  re¬ 
source  is  not  specified  in  the  TRP,  it  is  likely 
the  resource  will  not  be  available  for  the 
test.  The  TRP  is  revised  and  updated  on  a 
continuous  basis,  since  the  test  resource 
requirements  become  better  defined  as  the 
OT&E  plans  mature.  The  initial  TRP  serves 
as  a  baseline  for  comparison  of  planned 
OT&E  resources  with  actual  expenditures. 
Comparisons  of  the  initial  TRP  with  subse¬ 
quent  updates  provide  an  audit  trail  of 


changes  in  the  test  program  and  its  testing 
requirements.  The  AFOTEC  maintains  all 
TRPs  on  TRIMS;  this  permits  immediate 
response  to  all  queries  regarding  test  re¬ 
source  requirements. 

The  AFOTEC /RM  consolidates  the  re¬ 
source  requirements  from  all  TRPs  coordi¬ 
nating  with  participating  and  supporting 
organizations  and  agencies  outside 
AFOTEC,  Twice  yearly,  the  RM  office  pre¬ 
pares  a  draft  of  the  USAF  Program  for 
Operational  Test  (PO).  The  PO  is  a  master 
planning  and  programming  document  for 
resource  requirements  for  all  HQ  USAF- 
directed  OT&E  and  is  distributed  to  all 
concerned  commands,  agencies  and  orga¬ 
nizations  for  review  and  coordination.  It  is 
then  submitted  to  the  Air  Staff  for  review 
and  approval  by  the  Operational  Resource 
Management  Assessment  System  for  Test 
and  Evaluation  (ORMAS/TE),  which  op¬ 
erates  under  the  authority  of  HQ  AF/TE. 
The  ORMAS  Board  is  composed  of  HQ 
USAF  action  officers  and  senior  officers 
from  major  commands  (MAJCOMs)  and 
agencies  involved  in  OT&E;  it  meets  to 
resolve  impacts  and  conflicting  require¬ 
ments  at  the  appropriate  Air  Staff  level. 
Through  the  ORMAS  process,  HQ  USAF 
approves  the  PO,  which  becomes  a  direc¬ 
tive  to  participants  for  planning,  program¬ 
ming  and  budgeting  actions.  Agreements 
made  among  ORMAS  participants  regard¬ 
ing  TRP  and  PO  resource  requirements  are 
considered  binding. 

All  requests  for  test  resources  are  coordi¬ 
nated  by  HQ  AFOTEC  as  part  of  the  TRP 
preparation  process.  When  a  new  weapon 
system  development  is  first  identified, 
AFOTEC  provides  a  test  manager  (TM) 
who  begins  long-term  OT &E  planning.  The 
TM  begins  identifying  needed  test  re¬ 
sources,  such  as  instrumentation,  simula¬ 
tors  and  models,  and  works  with  the  re¬ 
sources  directorate  to  obtain  them.  If  the 
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required  resource  does  not  belong  to 
AFOTEC,  it  will  negotiate  with  the  com¬ 
mands  having  the  resource.  In  the  case  of 
models  and  simulators,  AFOTEC  surveys 
what  is  available,  assesses  credibility,  and 
then  coordinates  with  the  owner  or  devel¬ 
oper  to  use  it.  The  Joint  Technical  Coordi¬ 
nating  Group  publishes  a  document  on 
electronic  warfare  (EW)  models. 

Range  scheduling  should  be  done  early.  At 
least  a  year  is  required,  but  often  a  test  can 
be  accommodated  with  a  few  months'  no¬ 
tice  if  there  is  no  requirement  for  special 
equipment  or  modifications  to  be  provided 
at  the  range.  Some  of  the  Air  Force  ranges 
are  scheduled  well  in  advance  and  cannot 
accommodate  tests  that  encounter  delays 
or  retest  requirements. 

The  resource  manager  attempts  to  resolve 
conflicts  among  various  systems  compet¬ 
ing  for  scarce  test  resources  and  elevates 
the  request  to  the  Commander,  AFOTEC,  if 
necessary.  Decisions  on  resource  utiliza¬ 
tion  and  scheduling  are  based  on  the 
weapon  system's  assigned  priority. 

The  resource  manager  and  the  test  man¬ 
ager  also  arrange  for  use  of  the  resources  of 
other  Services,  non-DoD  government  agen¬ 
cies  and  contractors.  Use  of  non-U.S.  re¬ 
sources,  such  as  a  Canadian  range,  are  co¬ 
ordinated  by  Air  Force,  Chief  of  Staff/Di¬ 
rectorate  of  Test  and  Evaluation  (AF/TE) 
and  based  on  formal  Memoranda  of  Un¬ 
derstanding  (MOU).  The  U.S.  Air  Force- 
Europe/Diectorate  of  Operations-Opera- 
tions  (USAFE/DOQ)  handles  requests  for 
European  ranges.  Use  of  a  contractor- 
owned  resource,  such  as  a  model,  is  often 
obtained  through  the  System  Program  Of¬ 
fice  (SPO)  or  a  general  support  contract. 

15.4  TEST  RESOURCE  FUNDING 

The  Future  Years  Defense  Program  (FYDP), 
incorporating  a  biennial  budgeting  process. 


is  the  basic  DoD  programming  document 
that  records,  summarizes  and  displays  Sec¬ 
retary  of  Defense  (SECDEF)  decisions.  In 
the  FYDP,  costs  are  divided  into  three  cat¬ 
egories  for  each  acquisition  program  ele¬ 
ment:  R&D  costs,  investment  costs  and  op¬ 
erating  costs.  The  Congress  appropriates  to 
the  Office  of  Management  and  Budget 
(OMB),  and  OMB  apportions  funding 
through  the  SECDEF  to  the  Services  and  to 
other  defense  agencies.  The  Services  and 
defense  agencies  then  allocate  funds  to  oth¬ 
ers  (claimants,  subclaimants,  administer¬ 
ing  offices,  commanding  generals,  etc.). 

The  Planning,  Programming,  and  Budget¬ 
ing  System  (PPBS)  is  a  DoD  internal  system 
used  to  develop  input  to  the  Congress  for 
each  year's  budget  while  developing  fu¬ 
ture-year  budgets.  The  PPBS  is  calendar 
oriented.  There  are  concurrent  two-year 
PPBS  cycles  ongoing  at  one  time.  These 
cycles  are:  planning,  programming  and 
budgeting.  At  any  one  time  there  are  three 
budgets  being  worked  by  the  Services.  The 
current  two-year  budget  is  being  executed. 
The  next  six  years  of  defense  planning  is 
being  programmed,  and  long-range  pro¬ 
gram  plans  and  planning  guidance  are  be¬ 
ing  reviewed  for  updating. 

There  are  various  types  of  funding  in  the 
PPBS:  R&D  funding  for  maintaining  the 
technology  base;  exploratory  development 
funding  for  conducting  the  Concept  Explo¬ 
ration  Phase;  advanced  development  fund¬ 
ing  for  conducting  both  the  Concept  Explo¬ 
ration  Phase  and  the  Program  Definition 
and  Risk  Reduction  Phase;  engineering 
development  funding  for  conducting  the 
Engineering  and  Manufacturing  Develop¬ 
ment  Phase;  procurement  funding  for  con¬ 
ducting  the  Production ,  Fielding/Deploy¬ 
ment  and  Operational  Support  Phase. 
RDT&E  management  and  support  funding 
is  used  throughout  the  development  cycle 
until  the  system  is  operationally  deployed 
when  operations  and  maintenance  (O&M) 
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funding  is  used.  The  RDT&E  appropria¬ 
tion  funds  the  costs  associated  with  R&D, 
including  test  items,  DT&E  and  test  sup¬ 
port  of  OT&E  of  the  system  or  equipment 
and  the  test  items. 

Funding  that  is  planned,  programmed  and 
budgeted  through  the  PPBS  cycle  is  not 
always  the  same  funding  amount  that  the 
Congress  appropriates  or  the  PM  receives. 
If  the  required  funding  for  a  test  program  is 
not  authorized  by  the  Congress,  the  PM  has 
four  ways  to  react.  The  PM  can  submit  a 
supplemental  budget  (for  unfunded  por¬ 
tions  of  the  program),  request  deficiency 
funding  (for  unforeseen  program  problems) 
or  use  transfer  authority  (from  other  pro¬ 
grams  within  the  Service);  or  the  PM  can  try 
to  reprogram  the  needed  funds  (to  restruc¬ 
ture  the  program). 

Generally,  testing  that  is  accomplished  for 
a  specific  system  before  the  production 
decision  is  funded  from  RDT&E  appro¬ 
priations;  and  testing  that  is  accomplished 
after  the  production  decision  is  funded  from 
other  procurement  or  operations  and  main¬ 
tenance  appropriations.  Testing  of  product 
improvements,  block  upgrades  and  major 
modifications  is  funded  from  the  same  ap¬ 
propriations  as  the  program  development. 
Follow-on  Test  and  Evaluations  (FOT&E) 
are  normally  funded  from  O&M  funds. 

Funding  associated  with  T&E  (including 
instrumentation,  targets  and  simulations) 
are  identified  in  the  system  acquisition  cost 
estimates.  Service  acquisition  plans  and 
the  TEMP.  General  funding  information 
for  development  and  operational  tests  fol¬ 
lows: 

Development  Test  Funding.  Funds  required 
to  conduct  engineering  and  development 
tests  are  programmed  and  budgeted  by  the 
materiel  developer,  based  upon  the  re¬ 
quirements  of  the  TEMP.  These  costs  may 


include,  but  are  not  limited  to,  procuring 
test  samples/prototypes;  support  equip¬ 
ment;  transportation  costs;  technical  data; 
training  of  test  personnel;  repair  parts;  and 
test-specific  instrumentation,  equipment 
and  facilities.  The  DT&E  funds  are  ex¬ 
pended  for  contractor  and  government  de¬ 
velopmental  test  activities. 

The  Service  PM  may  be  required  to  pay  for 
the  use  of  test  resources,  such  as  the  MRTFB, 
and  for  the  development  of  specialized  re¬ 
sources  needed  specifically  for  testing  the 
weapon  system  being  developed. 

Operational  Test  (OT)  Funding.  Funds  re¬ 
quired  to  conduct  OT  are  usually  pro¬ 
grammed  and  budgeted  by  the  ^rvice 
operational  test  agency  or  organization. 
The  funds  are  programmed  in  the  Service's 
long-range  test  program,  and  the  funds 
requirements  are  obtained  from  the  test 
resourcing  documentation  and  TEMP. 

15.4.1  Army  Funding 

Test  resources  are  developed  and  funded 
under  various  Army  appropriations.  The 
Army  Materiel  Command  and  its  commod¬ 
ity  commands  provide  test  items,  spare 
parts,  support  items  (such  as  diagnostic 
equipment)  and  ammunition.  Soldiers, 
ranges,  fuel,  test  support  personnel  and 
maneuver  areas  are  provided  by  the  Train¬ 
ing  and  Doctrine  Command  (TRADOC)  or 
the  Army  Forces  Command  (FORSCOM). 
The  weapon  system  PM  uses  RDT &E  funds 
to  reimburse  these  supporting  commands 
for  costs  directly  related  to  his  test.  The 
weapon  system  materiel  developer  is  also 
responsible  for  funding  the  development 
of  new  test  resources  specifically  needed  to 
test  the  weapon  system.  Examples  of  such 
special-purpose  resources  include  mod¬ 
els,  simulations,  special  instrumentation 
and  test  equipment,  range  modifications, 
EW  simulators  and,  sometimes,  threat 
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simulators .  Although  the  Army  has  a  sepa¬ 
rate  budget  and  development  plan  for 
threat  simulators,  the  Army  Development 
and  Acquisition  of  Threat  Simulators 
(AD ATS)  program,  many  weapon  system 
developers  still  have  to  fund  the  cost  of 
new  threat  systems  that  are  specifically 
needed  to  test  their  weapon  system.  Army 
OPTEC  is  funded  through  the  PM's  pro¬ 
gram  element  and  is  given  direct  control  of 
OT&E  funds  for  each  program.  Funding 
requirements  are  developed  in  consonance 
with  the  Outline  Test  Plan. 

15.4.2  Navy  Funding 

In  the  Navy,  the  weapon  system  PM  is 
responsible  for  funding  the  development 
of  all  required  test-specific  resources  from 
the  program's  RDT&E  funds.  These  re¬ 
sources  include  test  articles,  expendables, 
one-of-a-kind  targets,  data  collection/re- 
duction  and  instrumentation.  The  devel¬ 
opment  of  generic  test  resources  that  can  be 
used  in  OT&E  of  multiple  weapon  systems 
such  as  targets,  threat  simulators  and  range 
capabilities,  is  funded  from  OPNAV  ge¬ 
neric  accounts  (such  as  target  development) 
and  not  from  weapon  systems  RDT&E.  The 
PM's  RDT&E  funds  pay  for  all  DT  and  OT 
through  OPEVAL.  The  PM  pays  for  all 
post-production  OT  with  program  funds. 

15.4.3  Air  Force  Funding 

In  the  Air  Force,  direct-cost  funding  re¬ 
quires  that  test-peculiar  (direct)  costs  asso¬ 
ciated  with  a  particular  test  program  be 
reimbursed  by  the  System  Program  Office 
to  the  designated  test  agency.  The  RDT&E 
appropriation  funds  the  cost  associated 
with  R&D,  including  test  items,  DT&E  and 
Air  Force  Materiel  Command  ( AFMC)  sup¬ 
port  of  OT&E  of  the  system  or  equipment 
and  the  test  items.  Costs  associated  with 
initial  operational  test  and  evaluation 


(lOT&E)  are  RDT&E  funded,  and  costs  of 
qualification  operational  test  and  evalua¬ 
tion  (QOT&E)  are  O&M  funded.  The 
AFOTEC  is  funded  through  its  own  pro¬ 
gram  element  and  has  direct  control  of 
OT&E  funds  for  all  programs.  The  lOT&E 
manager  prepares  a  TRP  that  summarizes 
the  resource  requirements  for  lOT&E  and 
related  test  support.  All  pretest  lOT&E  plan¬ 
ning  is  budgeted  through  and  paid  out  of 
the  O&M  appropriation.  The  FOT&E  costs 
are  paid  by  AFOTEC  and/ or  the  MAJCOM 
operating  the  system  and  funded  by  the 
O&M  appropriation. 

15.5  SUMMARY 

Test  resources  have  many  conflicting  de¬ 
mands  and  their  use  must  be  scheduled 
well  in  advance  of  a  test.  Resources  specific 
to  a  particular  test  must  often  be  developed 
and  funded  from  the  PM's  own  RDT&E 
budget.  Thus,  the  PM  and  his  testers  must 
ensure  that  test  resource  requirements  are 
identified  early  in  the  acquisition  cycle, 
that  they  are  documented  in  the  initial 
TEMP,  and  that  modifications  and  refine¬ 
ments  are  reported  in  the  TEMP  updates. 

Funds  for  testing  are  provided  by  congres¬ 
sional  appropriation  to  the  OMB,  which 
apportions  the  funds  to  the  Services  through 
the  SECDEF.  The  PPBS  is  the  DoD  process 
used  to  formulate  budget  requests  to  the 
Congress.  Requests  by  PMs  for  test  re¬ 
sources  are  usually  outlined  in  the  TEMP. 
Generally,  system  development  is  funded 
from  RDT&E  funds  until  the  system  is  op¬ 
erationally  deployed  and  maintained .  O&M 
funds  are  used  for  FOT&E  and  system  main¬ 
tenance.  The  weapon  system  materiel  de¬ 
veloper  is  also  responsible  for  funding  the 
development  of  new  test  resources  specifi¬ 
cally  needed  to  test  the  weapon  system.  The 
Air  Force  OTA  develops  and  directly  con¬ 
trols  OT&E  funds. 
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TEST  AND  EVALUATION  MASTER  PLAN 


16.1  INTRODUCTION 

Guidance  contained  in  Department  of  De¬ 
fense  (DoD)  5000. 2-R  stipulates  that  a  Test 
and  Evaluation  Master  Plan  (TEMP)  for¬ 
mat  shall  be  used  for  all  Acquisition  Cat¬ 
egory  (ACAT)  I  and  lA  or  Office  of  the 
Secretary  of  Defense  (OSD)  designated 
oversight  acquisition  programs.  This  rein¬ 
forces  the  philosophy  that  good  planning 
supports  good  operations.  For  effective  en¬ 
gineering  development  and  decision-mak¬ 
ing  processes,  an  overall  strategy  must  be 
developed  integrating  the  collection  and 
evaluation  of  test  data  on  required  perfor¬ 
mance  parameters.  Less  than  ACAT  I  pro¬ 
grams  are  encouraged  to  tailor  their  test 
and  evaluation  (T&E)  strategy  using  the 
TEMP  format  as  a  guide.  The  TEMP  relates 
program  schedule,  test  management  strat¬ 
egy  and  structure,  and  required  resources 
to:  critical  operational  issues;  critical  tech¬ 
nical  parameters;  minimum  acceptable  val¬ 
ues  (thresholds);  acquisition  strategy;  and, 
milestone  decision  points.  Feedback  about 
the  degree  of  system  performance  maturity 
and  its  operational  effectiveness  and  suit¬ 
ability  during  each  phase  is  essential  to  the 
successful  fielding  of  equipment  that  satis¬ 
fies  user  requirements. 

16.2  TEMP  DEVELOPMENT 

The  development  of  program  T&E  strat¬ 
egy,  codification  in  the  TEMP,  and  effective 
management  of  the  various  test  processes 
is  one  of  the  primary  functions  of  a  pro¬ 
gram  management  office  (PMO).  The  T&E 
strategy  is  highly  contingent  on  Phase  0 


concept(s)  that  are  deemed  appropriate 
for  satisfying  user  requirements.  As  out¬ 
lined  in  DoDD  5000.1,  part  1,  the  priority 
for  selecting  a  solution  is: 

(1)  a  non-materiel  solution,  such  as 
changes  to  tactics,  doctrine,  operational 
concepts,  training,  or  organization. 

(2)  the  sequence  of  materiel  alternatives 
is: 

(a)  use  or  modification  of  an  existing 
U.S.  military  system. 

(b)  use  or  modification  of  an  existing 
commercially  developed  or  Allied  system 
that  fosters  a  non-developmental  acquisi¬ 
tion  strategy. 

(c)  a  cooperative  research  and  de¬ 
velopment  program  with  one  or  more  Al¬ 
lied  nations. 

(d)  a  new  joint-Service  development 
program. 

(e)  a  new  Service-unique  develop¬ 
ment  program. 

The  quality  of  the  test  program  may  di¬ 
rectly  reflect  the  level  of  effort  expended  in 
its  development  and  execution.  This  varies 
in  direct  relationship  to  the  management 
imposed  by  the  program  manager  (PM) 
and,  to  some  extent,  by  the  system  engi¬ 
neer.  The  PM  must  evaluate  the  utility  of 


dedicated  T&E  staff  versus  matrix  support 
from  the  development  command.  The  lev¬ 
els  of  intensity  for  planning  and  executing 
T&E  fluctuate  with  changes  in  phases  of 
the  acquisition  process  and  in  T&E  staff 
support,  as  appropriate. 

Early  planning  of  long-range  strategies  can 
be  supported  with  knowledgeable  plan¬ 
ning  teams  (TE  Integrated  Product  Teams 
(IPT))  and  reviews  by  panels  of  senior  T&E 
management  officials  —  "gray  beards."  As 
the  tempo  of  actual  test  activities  begins  to 
build  (late  Phase  I  Program  Definition  and 
Risk  Reduction  (PDRR)  to  Phase  II  pre- 
LRIP  (low  rate  initial  production)  Engi¬ 
neering  and  Manufacturing  Development 
(EMD),  internal  T&E  management  staff  is 
needed  to  control  the  processes  and  evalu¬ 
ate  results. 

16.2.1  Program  Management  Office 
Responsibilities 

The  PMO  is  the  focal  point  of  the  develop¬ 
ment,  review  and  approval  process  for  the 
program  TEMP.  The  DoD  acquisition  pro¬ 
cess  requires  a  TEMP  as  one  of  the  primary 
management  strategy  documents  support¬ 
ing  the  decision  to  start  or  terminate  devel¬ 
opment  efforts  at  Milestone  I.  This  task  is  a 
"difficult  do"  during  Phase  0  since  some 
Services  do  not  formulate  or  staff  a  PMO 
until  program  start  (Milestone  (MS)  I).  An 
additional  complicating  factor  is  the  nebu¬ 
lous  condition  of  other  program  source 
documents  (Operational  Requirement 
Document  (ORD),  Technical  Management 
planning.  Acquisition  Strategy,  System 
Threat  Assessment,  Logistics  Support  Plan¬ 
ning,  etc.)  that  are  also  in  early  stages  of 
development/ updating  for  the  milestone 
review.  Since  the  TEMP  must  conform  to 
other  program  management  documents,  it 
frequently  lags  in  the  development  process 
and  does  not  receive  the  attention  needed 
from  PMO  or  matrix  support  personnel. 
Program  Management  Office  emphasis  on 


early  formulation  of  the  test  planning  teams 
(TE  IPT)  is  critical  to  the  successful  devel¬ 
opment  of  the  program  TEMP.  These  teams 
should  consist  of  the  requisite  players  so  a 
comprehensive  and  integrated  strategy 
compatible  with  other  engineering  and 
decision-making  processes  is  developed. 
The  PMO  will  find  that  the  number  of 
parties  desiring  coordination  on  the  TEMP 
far  exceed  the  "streamlined"  approval  pro¬ 
cess  signatories,  however,  it  must  be  coor¬ 
dinated.  An  early  start  in  getting  Service- 
level  concurrence  is  important  so  the  Mile¬ 
stone  Decision  document-submission 
schedule  can  be  supported  with  the  draft 
and  final  versions  of  the  TEMP.  Subse¬ 
quent  updates  do  not  become  easier,  as 
each  acquisition  phase  brings  new  plan¬ 
ning,  coordination  and  testing  require¬ 
ments. 

16.2.2  T&E  Planning 

Developing  an  overall  strategy  provides 
the  framework  for  incorporating  phase- 
oriented  T&E  activities  that  will  facilitate 
the  acquisition  process.  The  T&E  strategy 
should  be  consistent  with  the  program  ac¬ 
quisition  strategy,  identifying  requirements 
for  contractor  and  government  develop¬ 
ment  test  and  evaluation  (DT&E),  interac¬ 
tions  between  DT&E  and  operational  test 
and  evaluation  (OT&E),  and  provisions  for 
the  separate  initial  operational  test  and 
evaluation  (lOT&E).  An  evolutionary  ac¬ 
quisition  strategy  will  generally  include 
moderate  to  low-risk  technologies  that 
should  reduce  the  intensity  and  duration  of 
the  T&E  program.  It  does,  however,  in¬ 
clude  a  requirement  for  postproduction 
test  activities  as  the  system  is  modified  to 
accommodate  previously  unknown  new 
technologies,  new  threats  or  other  perfor¬ 
mance  enhancements. 

A  revolutionary  acquisition  strategy  in¬ 
corporates  all  the  latest  technologies  in 
the  final  production  configuration  and  is 
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generally  a  higher-risk  approach.  As  the 
contractor  works  on  maturing  emerging 
technologies,  the  T&E  workload  increases 
in  direct  proportion  to  the  difficulty  in  fix¬ 
ing  problems.  There  is  a  much  higher  po¬ 
tential  for  extended  schedules  with  itera¬ 
tive  test-fix-test  cycles. 

16.2.3  General  Test 

and  Evaluation  Planning  Issues 

The  Defense  Science  Board  (DSB)  (Refer¬ 
ence  41)  report  presented  guidance  on  T&E 
at  two  levels.  On  a  general  level  it  discussed 
a  number  of  issues  that  were  appropriate  to 
all  weapon  acquisition  programs.  These 
issues,  along  with  a  summary  discussion, 
are  given  below. 

16.2.3.1  Effects  of  Test  Requirements 
on  System  Acquisition 

The  acquisition  strategy  for  the  system 
should  allow  sufficient  time  between  the 
end  of  demonstration  testing  and  procure¬ 
ment,  as  contracted  with  limited  produc¬ 
tion  decisions,  to  allow  flexibility  for  modi¬ 
fication  of  plans  that  will  be  required.  It 
should  ensure  that  sufficient  dollars  are 
available  not  only  to  conduct  T&E  but  to 
allow  for  additional  T&E  that  is  always 
required  due  to  failure,  design  changes, 
etc.;  and,  it  should  be  evaluated  relative  to 
constraints  imposed  by: 

•  The  level  of  system  testing  at  various 
stages  of  the  research,  development,  test 
and  evaluation  (RDT&E)  cycle; 

•  The  number  of  test  items  available  and 
the  schedule  interface  with  other  systems 
needed  in  the  tests,  such  as  aircraft,  elec¬ 
tronics,  etc.; 

•  The  support  required  to  assist  in  pre¬ 
paring  for  and  conducting  tests  and  ana¬ 
lyzing  the  test  results; 


•  Being  evaluated  to  minimize  the  so- 
called  T&E  gap  caused  by  lack  of  hardware 
during  the  test  phase. 

16.2.3.2  Test  Requirements 
and  Restrictions 

Tests  should: 

•  Have  specific  objectives; 

•  List,  in  advance,  actions  to  be  taken  as 
a  consequence  of  the  test  results; 

•  Be  instrumented  to  permit  diagnosis  of 
the  cause  of  lack  of  performance  including 
random,  design  induced,  wear  out  and 
operator  error  failure; 

•  If  failures  occur,  not  be  repeated  with¬ 
out  a  detailed  analysis  of  the  failure.  ("Most 
likely  the  failure  will  not  go  away.") 

16.2.3.3  Trouble  Indicators 

Establish  an  early  detection  scheme  to  iden¬ 
tify  program  illness. 

When  a  program  begins  to  have  trouble, 
there  are  indicators  that  will  show  up  dur¬ 
ing  testing.  Some  of  these  indicators  are: 

•  A  test  failure; 

•  Any  repetitive  failure; 

•  A  revision  of  schedule  or  incremental 
funding  that  exceeds  the  original  plan; 

•  Any  relaxation  of  the  basic  require¬ 
ments  such  as  lower  performance. 

16.2.3.4  Requirement  For 
Test  Rehearsals 

Test  rehearsals  should  be  conducted  for 
each  new  phase  of  testing. 
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16.2.4  Scheduling 

Specific  issues  associated  with  test  sched¬ 
uling  are  listed  below. 

16.2.4.1  Building  Block  Test 
Scheduling 

The  design  of  a  set  of  tests  to  demonstrate 
feasibility  prior  to  the  EMD  Phase  should 
be  used.  This  will  allow  early  testing  of 
high-technical-risk  items,  and  subsequent 
tests  can  be  incorporated  into  the  hardware 
as  the  system  concept  has  been  demon¬ 
strated  as  feasible. 

16.2.4.2  Component 

and  Subsystem  Test  Plans 

Ensure  a  viable  component  and  subsystem 
test  plan.  Studies  show  that  almost  all  com¬ 
ponent  failures  will  be  the  kind  that  cannot 
be  easily  detected  or  prevented  in  full  sys¬ 
tem  testing.  System  failure  must  be  de¬ 
tected  and  fixed  in  the  component /sub¬ 
system  stage,  as  detecting  and  correcting 
failure  only  at  the  operational  test  level 
results  in  high  cost. 

16.2.4.3  Phasing  of  DT&E  and  lOT&E 

Problems  that  become  apparent  in  opera¬ 
tional  testing  can  often  be  evaluated  faster 
with  the  instrumented  development  test 
and  evaluation  (DT&E)  hardware.  The  in¬ 
tegrated  test  plan  should  provide  time  and 
money  to  investigate  test  failures  and  elimi¬ 
nate  causes  of  failures  before  other,  similar 
tests  take  place. 

16.2.4.4  Schedule  lOT&E  to  Include 
System  Interfaces  with  Other  Systems 

Whenever  possible,  the  initial  operational 
test  and  evaluation/ follow-on  operational 
test  and  evaluation  (lOT&E)/  (FOT&E)  of 
a  weapon  system  should  be  planned  to 


include  other  systems  that  must  have  a 
technical  interface  with  the  new  system. 
For  example,  the  missile  should  be  tested 
on  most  of  the  platforms  for  which  they  are 
programmed. 

The  preplanned  product  improvements 
(P^I)  strategy  is  a  variant  of  the  evolution¬ 
ary  development  process  in  which  you  rec¬ 
ognize  the  high-risk  technologies /sub¬ 
systems  and  put  them  on  a  parallel  devel¬ 
opment  track.  The  testing  strategy  should 
anticipate  the  requirements  to  evaluate  P^I 
item  maturity  and  then  test  the  system 
during  the  integration  of  the  additional 
capability. 

Advanced  Technology  Demonstrations 
( ATD)  may  provide  early  insights  into  avail¬ 
able  technologies  for  incorporation  into 
developmental  or  mature,  post-MS  III  sys¬ 
tems.  Using  proven,  mature  technology 
provides  a  lower-risk  strategy  and  may 
significantly  reduce  the  development  test¬ 
ing  work  load .  "To  assess  and  manage  risk, 
PMs  and  other  acquisition  managers  shall 
use  a  variety  of  techniques,  including  tech¬ 
nology  demonstrations,  prototyping,  and 
test  and  evaluation"  (DoDD  5000.1).  The 
process  for  verifying  contract  performance 
and  item  specifications,  testing  and  evalu¬ 
ation  of  threshold  performance  require¬ 
ments  in  the  ORD,  exit  criteria  or  the  ac¬ 
quisition  program  baseline  performance 
should  be  addressed  in  the  DT&E  strategy. 
The  DT&E  is  an  iterative  process  starting  at 
configuration  item/ software  module  lev¬ 
els  and  continuing  throughout  the  compo¬ 
nent  integration  into  subassemblies  and, 
finally,  system-level  performance  evalua¬ 
tions.  Operational  test  and  evaluation  is 
interwoven  into  early  DT&E  for  maximiz¬ 
ing  the  efficient  use  of  test  articles  and  test 
schedules.  However,  OT&E  must  remain  a 
distinct  thread  of  activity  that  does  not  lose 
its  identity  in  the  tapestry  of  test  events. 
Planning  for  test  resources  is  driven  by  the 
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sequence  and  intensity  of  development  test 
(DT)  and  operational  test  (OT)  events.  Re¬ 
source  coordination  is  an  equally  arduous 
task,  which  frequently  has  lead  times  equal 
to  major  program  development  activities. 
Included  in  the  program  T&E  strategy 
should  be  an  overshadowing  evaluation 
plan,  outlining  methodologies,  models, 
simulations  and  test  data  required  at  peri¬ 
odic  decision  points. 

The  TEMP  should;  (a)  address  critical  hu¬ 
man  issues  to  provide  data  to  validate  the 
results  of  human  factors  engineering  analy¬ 
ses;  and  (b)  require  identification  of  mis¬ 
sion  critical  operational  and  maintenance 
tasks. 

A  reliability  growth  (Test,  Analyze,  Fix 
and  Test  (TAFT))  program  should  be  de¬ 
veloped  to  satisfy  the  reliability  levels  re¬ 
quired  at  MS  III.  Reliability  tests  and  dem¬ 
onstrations  (DOD  3235. 1-H)  will  be  based 
on  actual  or  simulated  operational  condi¬ 
tions. 

Maintainability  will  be  verified  with  a 
maintainability  demonstration  (DoD 
3235.1-H)  before  MS  III. 

As  early  as  practicable,  developers  and 
test  agencies  will  assess  survivability  and 
validate  critical  survivability  character¬ 
istics  at  as  high  a  system  level  as  possible. 
The  TEMP  will  identify  the  means  by 
which  the  survivability  objectives  are 
validated. 

Field  engineering  test  facilities  and  test¬ 
ing  in  the  intended  operational  environ¬ 
ments  are  required  to  (1)  verify  electric  or 
electronic  systems  predicted  performance, 
(2)  establish  confidence  in  electromagnetic 
compatibility  design  based  on  standards 
and  specifications,  and  (3)  validate  electro¬ 
magnetic  compatibility  analysis  methodol¬ 
ogy- 


The  TEMP  will  address  health  hazard 
and  safety  critical  issues  to  provide  data 
to  validate  the  results  of  system  safety 
analyses. 

The  TEMP  strategy  should  directly  sup¬ 
port  the  development  of  more-detailed 
planning  and  resource  documents  needed 
to  execute  the  actual  test  events  and  subse¬ 
quent  evaluations. 

The  TEMP  shall  provide  a  road  map  for 
integrated  simulation,  test,  and  evaluation 
plans,  schedules,  and  resource  require¬ 
ments  necessary  to  accomplish  the  test  and 
evaluation  program.  Test  and  Evaluation 
planning  should  address  measures  of  ef¬ 
fectiveness/suitability  with  appropriate 
quantitative  criteria,  test  event  or  scenario 
description,  resource  requirements  and  test 
limitations.  Test  planning,  at  a  minimum, 
must  address  all  system  components  that 
are  critical  to  the  achievement  and  demon¬ 
stration  of  contract  technical  performance 
specifications  and  minimum  acceptable 
values  specified  in  the  ORD. 

16.3  TEMP  FORMAT 

The  format  specified  in  DoD  5000.2-R,  Ap¬ 
pendix  III,  is  required  for  all  acquisition 
category  I,  lA  and  OSD  designated  over¬ 
sight  programs  (Table  16-1).  It  may  be  tai¬ 
lored  as  needed  for  lesser  category  acquisi¬ 
tion  programs  at  the  discretion  of  the  mile¬ 
stone  decision  authority.  The  TEMP  is  in¬ 
tended  to  be  a  summary  document  outlin¬ 
ing  DT&E  and  OT&E  management  respon¬ 
sibilities  across  all  phases  of  the  acquisition 
process.  When  the  development  is  a  multi- 
Service  or  joint  acquisition  program,  one 
integrated  TEMP  is  developed  with  Ser¬ 
vice  annexes,  as  required.  A  Capstone 
TEMP  may  not  be  appropriate  for  a  single 
major  weapon  platform  but  could  be  used 
to  encompass  testing  of  a  collection  of  indi¬ 
vidual  systems,  each  with  its  own  annex 
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(e.g..  Ballistic  Missile  Defense  Organiza¬ 
tion,  Family  of  Tactical  Vehicles,  Forward 
Area  Air  Defense  Systems  (FAADS)).  A 
program  TEMP  is  updated  at  milestones, 
upon  program  baseline  breach  and  for  other 
significant  program  changes.  Updates  may 
consist  of  page  changes  and  are  no  longer 
required  when  a  program  has  no  further 
development  activities. 

The  TEMP  is  a  living  document  that  must 
address  changes  to  critical  issues  associ¬ 
ated  with  an  acquisition  program.  Major 
changes  in  program  requirements,  sched¬ 
ule  or  funding  usually  result  in  a  change  in 
the  test  program.  Thus,  the  TEMP  must  be 
reviewed  and  updated  on  program  change, 
on  basehne  breach  and  before  each  mile¬ 
stone  decision,  to  ensure  that  T&E  require¬ 
ments  are  current.  As  the  primary  docu¬ 
ment  used  in  the  OSD  review  and  mile¬ 
stone  decision  process  to  assess  the  ad¬ 
equacy  of  planned  testing  and  evaluation, 
the  TEMP  must  be  of  sufficient  scope  and 
content  to  explain  the  entire  T&E  program. 
The  key  topics  in  the  TEMP  are  shown  in 
Table  16-1. 

Each  TEMP  submitted  to  OSD  should  be  a 
summary  document,  detailed  only  to  the 
extent  necessary  to  show  the  rationale  for 
the  type,  amount  and  schedules  of  the  test¬ 
ing  plaimed.  It  must  relate  the  T&E  effort 
clearly  to  technical  risks,  operational  issues 
and  concepts,  system  performance,  reU- 
ability,  availability,  maintainability,  logis¬ 
tic  objectives  and  requirements,  and  major 
decision  points.  It  should  summarize  the 
testing  accomplished  to  date  and  explain 
the  relationship  of  the  various  models  and 
simulations,  subsystem  tests,  intepated 
system  development  tests  and  initial  op¬ 
erational  tests  that,  when  analyzed  in  com¬ 
bination,  provide  confidence  in  the  system's 
readiness  to  proceed  into  the  next  acquisi¬ 
tion  phase.  The  TEMP  must  address  the 
T&E  to  be  accomplished  in  each  program 
phase,  with  the  next  phase  addressed  in 


the  most  detail.  The  TEMP  is  also  used  as  a 
coordination  document  to  outline  each  test 
and  support  organization's  role  in  the  T&E 
program  and  identify  major  test  facilities 
and  resources.  The  TEMPs  supporting  the 
production  and  initial  deployment  deci¬ 
sion  must  include  the  T&E  planned  to  verify 
the  correction  of  deficiencies  and  to  com¬ 
plete  production  qualification  testing  and 
follow-on  OT&E. 

The  objective  of  the  OSD  TEMP  review 
process,  often  using  the  Automated  Test 
Planning  System  software,  is  to  ensure  suc¬ 
cessful  T&E  programs  that  will  support 
decisions  to  commit  resources  at  major 
milestones.  Some  of  the  T&E  issues  consid¬ 
ered  during  the  TEMP  review  process  in¬ 
clude: 

(1)  Are  DT&E  and  OT&E  initiated  early 
to  assess  performance,  identify  risks  and 
estimate  operational  potential? 

(2)  Are  critical  issues,  test  directives  and 
evaluation  criteria  related  to  mission  need 
and  operational  requirements  established 
well  before  testing  begins? 

(3)  Are  provisions  made  for  collecting 
sufficient  test  data  with  appropriate  test 
instrumentation  to  minimize  subjective 
judgment? 

(4)  Is  OT&E  conducted  by  an  organiza¬ 
tion  independent  of  the  developer  and  user? 

(5)  Do  the  test  methodology  and  instru¬ 
mentation  provide  a  mature  and  flexible 
network  of  resources  that  stress  (as  early  as 
possible)  the  weapon  system  in  a  variety  of 
realistic  environments? 

16.4  SUMMARY 

The  PMO  is  directly  responsible  for  the 
content  and  quality  of  the  test  strategy 
and  planning  document.  The  TEMP,  as 
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Table  16-1.  Test  and  Evaluation  Master  Plan  Format 


PARTI 

SYSTEM  INTRODUCTION 

MISSION  DESCRIPTION 

SYSTEM  THREAT  ASSESSMENT 

MEASURES  OF  EFFECTIVENESS  AND  SUITABILITY 

SYSTEM  DESCRIPTION 

CRITICAL  TECHNICAL  PARAMETERS 

PART  II 

INTEGRATED  TEST  PROGRAM  SUMMARY 

INTEGRATED  TEST  PROGRAM  SCHEDULE 

MANAGEMENT 

PART  III 

DEVELOPMENTTEST  AND  EVALUATION  OUTLINE 

DEVELOPMENT  TEST  AND  EVALUATION  OVERVIEW 

FUTURE  DEVELOPMENTAL  TEST  AND  EVALUATION 

PART  IV 

OPERATIONAL  TEST  AND  EVALUATION  OUTLINE 

OPERATIONAL  TEST  AND  EVALUATION  OVERVIEW 

CRITICAL  OPERATIONAL  ISSUES 

FUTURE  OPERATIONAL  TEST  AND  EVALUATION 

LIVE  FIRE  TEST  AND  EVALUATION 

PARTV 

TEST  AND  EVALUATION  RESOURCE  SUMMARY 

TEST  ARTICLES 

TEST  SITES  AND  INSTRUMENTATION 

TEST  SUPPORT  EQUIPMENT 

THREAT  REPRESENTATION 

TEST  TARGETS  AND  EXPENDABLES 

OPERATIONAL  FORCE  TEST  SUPPORT 

SIMULATIONS,  MODELS  AND  TEST  BEDS 

SPECIAL  REQUIREMENTS 

TEST  AND  EVALUATION  FUNDING  REQUIREMENTS 

MANPOWER/PERSONNEL  TRAINING 

APPENDIX  A 

BIBLIOGRAPHY 

APPENDIX  B 

ACRONYMS 

APPENDIX  C 

POINTS  OF  CONTACT 

ATTACHMENTS  (as  appropriate) 
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an  integrated  summary  management  tool, 
requires  an  extensive  commitment  of  man¬ 
hours  and  PM  guidance.  The  interactions 
of  the  various  T&E  players  and  support 
agencies  in  the  TE IPT  must  be  woven  into 
the  fabric  of  the  total  system  acquisition 


strategy.  Cost  and  schedule  implications 
must  be  negotiated  to  ensure  a  viable  T&E 
program  that  provides  timely  and  accurate 
data  to  the  engineering  and  management 
decision  makers. 
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MODULE 

Specialized  Testing 


The  nature  of  a  weapon  system  sometimes 
requires  the  use  of  a  specially  tailored  test 
and  evaluation  program.  In  some  cases, 
hazardous  testing  must  be  performed.  In 
other  cases,  testing  must  conducted  by  spe¬ 
cialized  organizations  or  at  special  times  in 
the  development  life  cycle. 

This  module  addresses  the  testing  of  spe¬ 
cial  weapons  (such  as  chemical,  laser  and 
space  systems),  embedded  computer  sys¬ 
tems,  electronic  war  fare  and  command- 
and-control  systems,  logistics  infrastruc¬ 
ture  test  and  evaluation,  and  production- 
related  testing  activities. 


SOFTWARE 
SYSTEMS  TESTING 


17.1  INTRODUCTION 

Software  development  presents  a  major 
development  risk  for  Acquisition  Cat¬ 
egory  (AC  AT)  I  and  I A  military  systems. 
Software  is  found  in  Automated  Infor¬ 
mation  Systems  (AIS)  and  weapon  sys¬ 
tem  software.  Software  systems,  such  as 
personnel  records  management  systems, 
financial  accounting  systems,  or  logistics 
records,  which  are  the  end  item  solution 
to  user  requirements  fall  in  the  AIS  cat¬ 
egory.  Performance  requirements  for  the 
AIS  typically  drive  the  host  hardware 
configurations  and  are  managed  by  the 
Major  Automated  Information  Systems 
Review  Council  (MAISRC)  chaired  by  the 
Assistant  Secretary  of  Defense  (Com¬ 
mand,  Control,  Communications  and  In¬ 
telligence  (C^I).  The  Director,  Operational 
Test  and  Evaluation  (DOTE)  and  the  Di¬ 
rector,  Test,  Systems  Engineering,  and 
Evaluation  (DTSEE)  are  principal  mem¬ 
bers  of  the  MAISRC.  Software  develop¬ 
ments,  such  as  avionics  systems,  weap¬ 
ons  targeting  and  control,  and  navigation 
computers,  that  are  a  subset  of  the  hard¬ 
ware  solution  to  user  requirements  fall  in 
the  weapon  system  software  category. 
Performance  requirements  for  the  sys¬ 
tem  hardware  are  flowed  down  to  drive 
the  functionality  of  the  software  resident 
in  onboard  computers.  The  effectiveness 
of  the  weapon  system  software  is  re¬ 
viewed  as  part  of  the  overall  system  re¬ 
view  by  the  Defense  Acquisition  Board 
(DAB)  chaired  by  the  Under  Secretary  of 
Defense  for  Acquisition  and  Technology 


USD  (A&T).  The  DOTE  is  a  principal 
member  and  the  DTSEE  is  an  advisor  to 
the  DAB.  Software  development  histori¬ 
cally  has  escalated  the  cost  and  reduced 
the  reliability  of  weapon  systems.  Em¬ 
bedded  computer  systems  that  are  physi¬ 
cally  incorporated  into  larger  weapon  sys¬ 
tems,  have  a  major  function  of  data  pro¬ 
cessing.  The  output  of  the  systems  are 
normally  information,  control  signals  or 
computer  data  required  by  the  host  sys¬ 
tem  to  complete  its  mission.  Although 
hardware  and  software  often  contribute 
in  equal  measure  to  successful  imple¬ 
mentation  of  system  functions,  there  have 
been  relative  imbalances  in  their  treat¬ 
ment  during  system  development. 

Automated  Information  Systems,  once  de¬ 
veloped,  integrated  and  tested  in  the  host 
hardware  are  essentially  ready  for  produc¬ 
tion.  Software  in  weapon  systems,  once 
integrated  in  the  host  hardware,  continue 
to  be  tested  as  a  component  of  the  total 
system  and  is  not  ready  for  production 
until  the  total  system  has  successfully  dem¬ 
onstrated  required  performance.  Any 
changes  to  weapon  system  hardware  con¬ 
figuration  may  stimulate  changes  to  the 
software.  The  development  of  all  software 
systems  involves  a  series  of  activities  in 
which  there  are  frequent  opportunities  for 
errors.  Errors  may  occur  at  the  inception  of 
the  process,  when  the  requirements  may  be 
erroneously  specified,  or  later  in  the  devel¬ 
opment  cycle,  when  system  integration  is 


implemented.  This  chapter  addresses  the 
use  of  testing  to  obtain  insights  into  the 
development  risk  of  AIS  and  weapon  sys¬ 
tem  software,  particularly  as  it  pertains  to 
the  software  development  processes. 

17.2  DEFINITIONS 

The  term  Automated  Information  System 
(AIS)  is  defined  in  Department  of  Defense 
(DoD)  5000.2-R  as  a  combination  of  com¬ 
puter  hardware  and  software,  data,  or  tele¬ 
communications,  that  performs  functions 
such  as  collecting,  processing,  transmit¬ 
ting,  and  displaying  information.  Excluded 
are  computer  resources,  both  hardware 
and  software,  that  are:  physical  part  of, 
dedicated  to,  or  essential  in  real  time  to  the 
mission  performance  of  weapon  systems. 
(There  is  some  indication  that  DoD  Direc¬ 
tive  (DoDD)  8000.1,  Defense  Information 
Management  Program  providing  guidance 
on  ATS  development,  will  be  incorporated 
in  a  future  change  to  the  5000  series  on 
acquisition  management.) 

The  term  weapon  system  software  includes 
automated  data  processing  equipment,  soft¬ 
ware  or  services;  and  the  function,  opera¬ 
tion  or  use  of  the  equipment  software  or 
services  involves: 

(1)  Intelligence  activities; 

(2)  Cryptologic  activities  related  to  na¬ 
tional  security; 

(3)  Command  and  control  of  military 
forces; 

(4)  Equipment  that  is  an  integral  part  of  a 
weapons  system; 

(5)  Critical,  direct  fulfillment  of  military 
or  intelligence  missions. 

Acquisition  of  software  for  DoD  is  de¬ 
scribed  in  Mil-Std-498,  which  has  been 


waiver  for  use  until  commercial  standards 
such  as  EIA  640  or  J-Std-016  become  the 
guidance  for  software  development. 
Guidance  may  also  be  found  in  DoD 
5000.2-R. 

17.3  PURPOSE  OF  SOFTWARE 
TEST  AND  EVALUATION 

A  major  problem  in  software  development 
is  a  lack  of  well-defined  requirements.  If 
requirements  are  not  well-defined,  errors 
can  multiply  throughout  the  development 
process.  As  illustrated  in  Figure  17-1,  er¬ 
rors  may  occur  at  the  inception  of  the  pro¬ 
cess.  These  errors  may  occur  during  re¬ 
quirements  definition,  when  objectives  may 
be  erroneously  or  imperfectly  specified; 
during  the  later  design  and  development 
stages,  when  these  objectives  are  imple¬ 
mented;  and  during  software  maintenance 
and  operational  phases,  when  software 
changes  are  needed  to  eliminate  errors  or 
enhance  performance.  Estimates  of  in¬ 
creased  software  costs  arising  from  incom¬ 
plete  testing  help  to  illustrate  the  dimen¬ 
sion  of  software  life-cycle  costs.  Averaged 
over  the  operational  life  cycle  of  a  com¬ 
puter  system,  development  costs  encom¬ 
pass  approximately  30  percent  of  total  sys¬ 
tem  costs.  The  remaining  70  percent  of  life- 
cycle  costs  are  associated  with  maintenance, 
which  includes  system  enhancements  and 
error  correction.  Complete  testing  during 
earlier  development  phases  may  have  de¬ 
tected  these  errors.  The  relative  costs  of 
error  correction  increase  as  a  function  of 
time  from  the  start  of  the  development 
process.  Relative  costs  of  error  correction 
rise  dramatically  between  requirements  and 
design  phases  and  more  dramatically  dur¬ 
ing  code  implementation  (Figure  17-1). 

Previous  research  in  the  area  of  software 
test  and  evaluation  (T&E)  reveals  that  half 
of  all  maintenance  costs  are  incurred  in  the 
correction  of  previously  undetected  errors. 
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Figure  17-1.  The  Error  Avalanche 


Approximately  one-half  of  the  operational 
life-cycle  costs  can  be  traced  directly  to 
inadequate  or  incomplete  testing  activities. 
In  addition  to  cost  increases,  operational 
implications  of  software  errors  in  weapon 
systems  can  result  in  mission  critical  soft¬ 
ware  failures  that  may  impact  mission  suc¬ 
cess  and  personnel  safety. 

A  more  systematic  and  rigorous  approach 
to  software  testing  is  required.  To  be  effec¬ 
tive,  this  approach  must  be  applied  to  all 
phases  of  the  development  process  in  a 
planned  and  coordinated  manner,  begin¬ 
ning  at  the  earliest  design  stages  and  pro¬ 
ceeding  through  operational  testing  of  the 
integrated  system.  Early,  detailed  software 
T&E  planning  is  critical  to  the  successful 
development  of  a  computer  system. 

17.4  SOFTWARE  DEVELOPMENT 
PROCESS 

Software  engineering  technologies  used  to 
produce  operational  software  are  key  risk 
factors  in  a  development  program.  The  T&E 
program  should  help  determine  which  of 
these  technologies  increase  risk  and  have  a 
life-cycle  impact.  A  principal  source  of  risk 
is  the  support  software  required  to  develop 
operational  software.  In  terms  of  life-cycle 
impact,  operational  software  problems  are 
commonly  associated  with  the  difficulty  in 
maintaining  and  supporting  the  software 
once  deployed.  Software  assessment  re¬ 
quires  an  analysis  of  the  life-cycle  impact, 
which  varies  depending  on  the  technology 
used  to  design  and  implement  the  soft¬ 
ware.  One  approach  to  reducing  long-term 
life-cycle  risks  is  to  use  ADA  language  and 
common  hardware  throughout  the  devel¬ 
opment  and  operation  of  the  software. 
ITiese  life-cycle  characteristics  that  affect 
operational  capabilities  must  be  ad¬ 
dressed  in  the  Test  and  Evaluation  Mas¬ 
ter  Plan  (TEMP),  and  tests  should  be  de¬ 
veloped  to  identify  problems  caused  by 


these  characteristics.  The  technology  used 
to  design  and  implement  the  software 
may  significantly  affect  software  support- 
ability  and  maintainability. 

The  TEMP  must  sufficiently  describe  the 
acceptance  criteria  or  software  maturity 
metrics  from  the  written  specifications  that 
will  lead  to  operational  effectiveness  and 
suitability.  The  specifications  must  define 
the  required  software  metrics  to  set  objec¬ 
tives  and  thresholds  for  mission  critical 
functions.  Additionally,  these  metrics 
should  be  evaluated  at  the  appropriate  stage 
of  system  development  rather  than  at  some 
arbitrarily  imposed  milestone. 

17.5  T&E  IN  THE  SOFTWARE 
LIFE  CYCLE 

Software  testing  is  an  iterative  process  ex¬ 
ecuted  at  all  development  stages  to  exam¬ 
ine  program  design  and  code  to  exp>ose 
errors.  Software  test  planning  should  be 
described  in  the  TEMP  with  the  same  care 
as  test  plaiming  for  other  system  compo¬ 
nents  (Figure  17-2, 17-3). 

17.5.1  Testing  Approach 

The  integration  of  software  development 
into  the  overall  acquisition  process  dic¬ 
tates  a  testing  process  consistent  with  the 
bottom-up  approach  taken  with  hardware 
development.  The  earliest  stage  of  soft¬ 
ware  testing  is  characterized  by  heavy 
human  involvement  in  basic  design  and 
coding  processes.  Thus,  human  testing  is 
defined  as  informal,  non-computer-based 
methods  of  evaluating  architectures,  de¬ 
signs  and  interfaces.  It  can  consist  of: 

•  Inspections  —  The  programmer  ex¬ 
plains  his/her  work  to  a  small  group  of 
peers  with  discussion  and  direct  feed¬ 
back  on  errors,  inconsistencies  and  omis¬ 
sions. 
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Figure  1 7-2.  System  Development  Process 
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•  Walk-through  —  A  group  of  peers 
develop  test  cases  to  evaluate  work  to  date 
and  give  direct  feedback  to  the  program¬ 
mer. 

•  Desk  Checking  —  A  self  evaluation  is 
made  by  the  programmer  of  his/her  own 
work.  There  is  a  low  probability  of  identify¬ 
ing  his/her  errors  of  logic  or  coding. 

•  Peer  Ratings  —  Mutually  supportive, 
anonymous  reviews  are  performed  by 
groups  of  peers  with  collaborative  evalua¬ 
tions  and  feedback. 

•  Design  Reviews — Preliminary  design 
reviews  (PDRs)  and  critical  design  reviews 
(CDRs)  provide  milestones  in  the  develop¬ 
ment  efforts  that  review  development  and 
evaluations  to  date.  An  independent  verifi¬ 
cation  and  validation  (IV&V)  contractor 
may  facilitate  the  government's  ability  to 
give  meaningful  feedback. 

Once  the  development  effort  has  matured 
beyond  the  benefits  of  human  testing,  com¬ 
puterized-software-only  testing  may  be 
appropriate.  It  is  performed  to  determine 
the  functionality  of  the  software  when  tested 
as  an  entity  or  "build."  Documentation  con¬ 
trol  is  essential  so  that  test  results  are  corre¬ 
lated  with  the  appropriate  version  of  the 
build.  Software  testing  is  usually  conducted 
using  some  combination  of  "blackbox"  and 
"white  box"  testing. 

•  Black  Box  —  Functional  testing  of  a 
software  unit  without  knowledge  of  how 
the  internal  structure  or  logic  will  process 
the  input  to  obtain  the  specified  output. 
Within-boundary  and  out-of-boundary 
stimulants  test  the  software’s  ability  to 
handle  abnormal  events.  Most  likely  cases 
are  tested  to  provide  a  reasonable  assur¬ 
ance  that  the  software  will  demonstrate 
specified  performance.  Even  the  simplest 
software  designs  rapidly  exceed  our  capac¬ 
ity  to  test  all  alternatives. 


•  White  Box  Structural  testing  of  the 
internal  logic  and  software  structure  pro¬ 
vides  an  opportunity  for  more  extensive 
identification  and  testing  of  critical  paths. 
The  process  and  objectives  are  otherwise 
very  similar  to  black  box  testing. 

Testing  should  be  performed  from  the  bot¬ 
tom  up.  The  smallest  controlled  software 
modules — computer  software  units — are 
tested  individually.  They  are  then  com¬ 
bined  or  integrated  and  tested  in  larger 
aggregate  groups  or  builds.  When  this  pro¬ 
cess  is  complete,  the  software  system  is 
tested  in  its  entirety.  Obviously,  as  errors 
are  found  in  the  latter  stages  of  the  test 
program,  a  return  to  earlier  portions  of  the 
development  program  to  provide  correc¬ 
tions  is  required.  The  cost  impact  of  error 
detection  and  correction  can  be  diminished 
using  the  bottom-up  testing  approach. 

System  level  testing  can  begin  once  all 
modules  in  the  computer  software  con¬ 
figuration  item  (CSCI)  have  been  coded 
and  individually  tested.  A  software  inte¬ 
gration  lab  (SIL),  with  adequate  machine 
time  and  appropriate  simulations,  will  fa¬ 
cilitate  hardware  simulation /emulation 
and  the  operating  environment.  If  data 
analysis  indicates  proper  software  func¬ 
tioning,  it  is  time  to  advance  to  a  more 
complex  and  realistic  test  environment. 

•  Hot  Bench  Testing — Integration  of  the 
software  released  from  the  SIL  for  full-up 
testing  with  actual  system  hardware  in  a 
hardware-in-the-loop  (HWIL)  facility 
marks  a  significant  advance  in  the  develop¬ 
ment  process.  Close  approximation  of  the 
actual  operating  environment  should  pro¬ 
vide  test  sequences  and  stress  needed  to 
evaluate  the  effectiveness  of  the  software 
system(s).  Problems  stimulated  by  the 
"noisy  environment,"  interface  problems, 
electromagnetic  interference  (EMI)  and  dif¬ 
ferent  electrical  transients  should  surface. 
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Good  hardware  and  software  test  programs 
leading  up  to  HWIL  testing  should  aid  in 
isolating  problems  to  the  hardware  or  soft¬ 
ware  side  of  the  system.  Caution  should  be 
taken  to  avoid  any  outside  stimuli  that 
might  trigger  unrealistic  responses. 

•  Field  Testing  —  Development  test  and 
evaluation  (DT&E)  and  operational  test  and 
evaluation  (OT&E)  events  must bedesigned 
to  provide  for  data  collection  processes  and 
instrumentation  that  will  measure  system 
responses  and  allow  data  analysts  to  iden¬ 
tify  the  appropriate  causes  of  malfunctions. 
Field  testing  should  be  rigorous,  providing 
environmental  stresses  and  mission  pro¬ 
files  likely  to  be  encountered  in  operational 
scenarios.  Government  software  support 
facilities  tasked  for  future  maintenance  of 
the  software  system  should  be  brought  on 
board  to  familiarize  them  with  the  system 
operating  characteristics  and  documenta¬ 
tion.  Their  expertise  should  be  included  in 
the  software  T&E  process  to  assist  in  the 
selection  of  stimuli  likely  to  expose  soft¬ 
ware  problems. 

It  is  critical  that  adequate  software  T&E 
information  be  contained  in  documents 
such  as  TEMPs  and  test  plans.  The  TEMP 
must  define  characteristics  of  critical  soft¬ 
ware  components  that  effectively  address 
objectives  and  thresholds  for  mission  criti¬ 
cal  functions.  The  measures  of  effective¬ 
ness  (MOEs)  must  support  the  critical  soft¬ 
ware  issues.  The  test  plan  should  specify 
the  test  methodologies  that  will  be  applied. 
Test  methodologies  consist  of  two  compo¬ 
nents.  The  first  is  the  test  strategy  that 
guides  the  overall  testing  effort,  and  the  sec¬ 
ond  is  the  testing  technique  that  is  applied 
within  the  framework  of  a  test  strategy. 

Effective  test  methodologies  require  real¬ 
istic  software  test  environments  and  sce¬ 
narios.  The  test  scenarios  must  be  ap¬ 
propriate  for  the  test  objectives;  i.e.,  the  test 


results  must  be  interpretable  in  terms  of 
software  test  objectives.  The  test  scenarios 
and  analysis  should  actually  verify  and 
validate  accomplishment  of  requirements. 
The  test  environments  must  be  chosen  on  a 
careful  analysis  of  characteristics  to  be  dem¬ 
onstrated  and  its  relationship  to  the  devel¬ 
opment,  operational  and  support  environ¬ 
ments.  In  addition,  environment  must  be 
representative  of  that  in  which  the  soft¬ 
ware  will  be  maintained. 

17.5.2  Independent  Verification 
and  Validation 

Independent  verification  and  validation  are 
risk-reducing  techniques  that  are  applied 
to  major  software  development  efforts.  The 
primary  purpose  of  IV&V  is  to  ensure  that 
software  meets  requirements  and  is  reli¬ 
able  and  maintainable.  The  IV&V  is  effec¬ 
tive  only  if  implemented  early  in  the  soft¬ 
ware  development  schedule.  Requirements 
analysis  and  risk  assessment  are  the  most 
critical  activities  performed  by  IV&V  orga¬ 
nizations;  their  effectiveness  is  limited  if 
brought  on  board  a  project  after  the  fact. 
Often,  there  is  a  reluctance  to  implement 
rV&V  because  of  the  costs  involved,  but 
early  implementation  of  IV&V  will  result 
in  lower  overall  costs  of  error  correction 
and  software  maintenance.  As  develop¬ 
ment  efforts  progress,  IV&V  involvement 
typically  decreases.  This  is  due  more  to  the 
expense  of  continued  involvement  than  to 
a  lack  of  need.  For  an  IV&V  program  to  be 
effective,  it  must  be  the  responsibility  of  an 
individual  or  organization  external  to  the 
software  development  program  manager 
(PM). 

The  application  of  the  IV&V  process  to 
software  development  maximizes  the  main¬ 
tainability  of  the  fielded  software  system, 
while  minimizing  the  cost  of  developing 
and  fielding  it.  Maintenance  of  a  software 
system  falls  into  several  major  categories: 
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corrective  maintenance,  modifying  soft¬ 
ware  to  correct  errors  in  operation;  adap¬ 
tive  maintenance,  modifying  the  software 
to  meet  changing  requirements;  and  per¬ 
fective  maintenance,  modifying  the  soft¬ 
ware  to  incorporate  new  features  or  im¬ 
provements. 

The  IV&V  process  maximizes  the  reli¬ 
ability  of  the  software  product,  which 
eases  the  performance  of  and  minimizes 
the  need  for  corrective  maintenance.  It 
attempts  to  maximize  the  flexibility  of 
the  software  product,  which  eases  the 
performance  of  adaptive  and  perfective 
maintenance.  These  goals  are  achieved 
primarily  by  determining  at  each  step  of 
the  software  development  process  that 
the  software  product  completely  and  cor¬ 
rectly  meets  the  specific  requirements 
determined  at  the  previous  step  of  devel¬ 
opment.  This  step-by-step,  iterative  pro¬ 
cess  continues  from  the  initial  definition 
of  system  performance  requirements 
through  final  acceptance  testing. 

The  review  of  software  documentation  at 
each  stage  of  development  is  a  major  por¬ 
tion  of  the  verification  process.  The  cur¬ 
rent  documentation  is  a  description  of 
the  software  product  at  the  present  stage 
of  development  and  will  define  the  re¬ 
quirements  laid  on  the  software  product 
at  the  following  stage.  Careful  examina¬ 
tion  and  analysis  of  the  development 
documentation  ensure  that  each  step  in 
the  software  design  process  is  consistent 
with  the  previous  step.  Omissions,  in¬ 
consistencies  or  design  errors  can  then  be 
identified  and  corrected  early  in  the  de¬ 
velopment  process. 

Continuing  participation  in  formal  and  in¬ 
formal  design  reviews  by  the  IV & V  organi¬ 
zation  maintains  the  communication  flow 
between  software  system  "customers"  and 
developers,  ensuring  that  software  design 


and  production  proceed  with  minimal 
delays  and  misunderstandings.  Frequent 
informal  reviews,  design  and  code  walk¬ 
through  and  audits  ensure  that  the  pro¬ 
gramming  standards,  software  engineer¬ 
ing  standards,  software  quality  assurance 
and  configuration  management  proce¬ 
dures  designed  to  produce  a  reliable, 
maintainable  operational  software  sys¬ 
tem  are  followed  throughout  the  process. 
Continuous  monitoring  of  computer 
hardware  resource  allocation  through¬ 
out  the  software  development  process 
also  ensures  that  the  fielded  system  has 
adequate  capacity  to  meet  operation  and 
maintainability  requirements. 

The  entire  testing  process,  from  the  plan¬ 
ning  stage  through  final  acceptance  test,  is 
also  approached  in  a  step-by-step  manner 
by  the  IV&V  process.  At  each  stage  of  de¬ 
velopment,  the  functional  requirements 
determine  test  criteria  as  well  as  design 
criteria  for  the  next  stage.  An  important 
function  of  the  IV&V  process  is  to  ensure 
that  the  test  requirements  are  derived  di¬ 
rectly  from  the  performance  requirements 
and  are  independent  of  design  implemen¬ 
tation.  Monitoring  of,  participation  in  and 
performance  of  the  various  testing  and  in¬ 
spection  activities  by  the  IV&V  contractor 
ensure  that  the  developed  software  meets 
requirements  at  each  stage  of  development. 

Throughout  the  software  development 
process,  the  IV&V  contractor  reviews  any 
proposals  for  software  enhancement  or 
change,  proposed  changes  in  develop¬ 
ment  base-  lines,  and  proposed  solutions 
to  design  or  implementation  problems  to 
ensure  that  the  original  performance  re¬ 
quirements  are  not  forgotten.  An  impor¬ 
tant  facet  of  the  FV&V  contractor’s  role  is 
to  act  as  the  objective  third  party,  con¬ 
tinuously  maintaining  the  "audit  trail" 
from  the  initial  performance  requirements 
to  the  final  operational  system. 
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17.6  SUMMARY 

There  is  a  useful  body  of  software  testing 
technologies  that  can  be  applied  to  test¬ 
ing  of  AIS  and  weapon  system  software. 
As  a  technical  discipline,  though,  software 
testing  is  still  maturing.  There  is  a  growing 
foundation  of  guidance  documents  to  guide 
the  PM  in  choosing  one  testing  technique 
over  another.  One  example  is  the  USAF 
Software  Technology  Support  Center's 
Guidelines  for  Successful  Acquisition  and 
Management  of  Software-intensive  Sys¬ 
tems.  The  Air  Force  Operational  Test  and 
Evaluation  Center  has  also  developed  a 


course  on  Software  OT&E.  It  is  apparent 
that  systematic  T&E  techniques  are  far  su¬ 
perior  to  ad-hoc  testing  techniques.  Imple¬ 
mentation  of  an  effective  software  T&E 
plan  requires  a  set  of  strong  technical  and 
management  controls.  Given  the  increas¬ 
ing  amount  of  AIS  and  weapon  system 
software  being  acquired,  there  will  be  an 
increased  emphasis  on  tools  and  techniques 
for  software  T&E.  For  more-detailed  infor¬ 
mation  on  weapon  system  software  devel¬ 
opment  and  testing,  review  the  Defense 
Systems  Management  College's  Mission 
Critical  Computer  Resource  Management 
Guide. 


17-10 


TESTING  FOR  VULNERABILITY 
AND  LETHALITY 


18.1  INTRODUCTION 

This  chapter  addresses  the  need  to  explore 
the  vulnerability  and  lethality  aspects  of  a 
system  through  test  and  evaluation  (T&E) 
practices  and  procedures .  In  particular,  this 
chapter  describes  the  legislatively-man¬ 
dated  Live  Fire  Test  Program,  which  has 
been  established  to  evaluate  the  survivabil¬ 
ity  and  lethality  of  developing  systems. 
(Table  18-1)  It  also  discusses  the  role  of  T&E 
in  assessing  a  system's  ability  to  perform  in 
a  nuclear  combat  environment.  The  discus¬ 
sion  of  testing  for  nuclear  survivability  is 
based  primarily  on  information  contained 
in  the  "Nuclear  Survivability  Handbook 
for  OT&E,"  prepared  by  the  Air  Force  Op¬ 
erational  Test  and  Evaluation  Center  (Refer¬ 
ence  91). 

18.2  LIVE  FIRE  TESTING 
18.2.1  Background 

In  March  1 984,  the  Office  of  the  Secretary  of 
Defense  (OSD)  chartered  a  joint  T&E  pro¬ 
gram  designated  "The  Joint  Live  Fire  Pro¬ 
gram."  This  program  was  to  assess  the  vul¬ 
nerabilities  and  lethality's  of  selected  U.S. 
and  threat  systems  already  fielded.  The 
controversy  over  joint  live  fire  testing  of  the 
Army’s  Bradley  Fighting  Vehicle  System, 
subsequent  congressional  hearings  and 
media  exposure  resulted  in  provisions  be¬ 
ing  incorporated  in  the  National  Defense 
Authorization  Act  of  FY87.  This  act  re¬ 
quired  an  OSD-managed  Live  Fire  Test¬ 
ing  (LFT)  program  for  major  acquisition 


programs  fitting  certain  criteria.  Subsequent 
amendments  to  legislative  guidance  have 
dictated  the  current  program.  The  Depart¬ 
ment  of  Defense  (DoD)  implementation  of 
congressional  guidance  in  DoD  5000.2-R, 
requires  that  "covered  systems,  major  mu¬ 
nitions  programs,  missile  programs,  or 
product  improvements  to  these "  (i.e..  Ac¬ 
quisition  Category  (ACAT)  I  and  n  pro¬ 
grams)  must  execute  survivability  and  le¬ 
thality  testing  before  Milestone  (MS)  III, 
full-rate  production.  Additionally,  post¬ 
production  product  improvements  to  those 
systems  may  reinitiate  LFT  requirements. 
The  Secretary  of  Defense  has  delegated  the 
authority  to  waive  requirements  for  the 
full-up,  system  level  Live  Fire  Test  and 
Evaluation  (LFT&E)  before  the  system 
passes  MS  II,  to  the  Under  Secretary  of 
Defense  (Acquisition  and  Technology 
(USD(A&T))  for  ACAT  ID  and  the  CAE  for 
ACAT  II  programs,  when  it  would  be  un¬ 
reasonably  expensive  or  impractical.  An 
alternative  vulnerability  and  lethality  T&E 
program  must  still  be  accomplished.  Pro¬ 
grams  subject  to  LFT  or  designated  for 
oversight  are  listed  on  the  OSD  annual  T &E 
oversight  list.  The  DoD  agent  for  manage¬ 
ment  of  the  Live  Fire  Test  program  is  the 
Director,  Operational  Test  and  Evaluation 
(DOTE).  This  type  of  development  test 
and  evaluation  (DT&E)  must  be  planned 
to  start  early  enough  in  the  development 
process  to  impact  design  and  to  provide 
timely  test  data  for  the  OSD  Live  Fire  Test 
Report  required  for  the  MS  III  Defense 


Table  18-1.  Relationships  Between  Key  Concepts 
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Acquisition  Board  (DAB)  and  congressional 
committees.  The  Service-detailed  Live  Fire 
Test  Plan  must  be  reviewed  and  approved 
by  the  DOTE,  and  LFT  must  be  addressed 
in  Part  IV  of  the  program  Test  and  Evalua¬ 
tion  Master  Plan  (TEMP).  The  OSD  had 
previously  published  guidelines,  elements 
of  which  have  subsequently  been  incorpo¬ 
rated  into  the  latest  revision  to  the  5000 
series  (DoD  5000.2-R,  Appendix  IV). 

18.2.2  Live  Fire  Tests 

There  are  varying  types  and  degrees  of  live 
fire  tests.  The  matrix  in  Table  18-2  illus¬ 
trates  the  various  possible  combinations. 
Full-scale,  full-up  testing  is  usually  consid¬ 
ered  to  be  the  most  realistic  and  is  the  type 
of  testing  called  for  in  the  National  Defense 
Authorization  Act  for  FY87. 

The  importance  of  full-scale  testing  has 
been  well  demonstrated  by  the  Joint  Live 
Fire  0LF)  tests.  In  one  case,  these  tests 
contradicted  earlier  conclusions  concern¬ 
ing  the  flammability  of  a  new  hydraulic 
fluid  used  in  F-15  and  F-1 6  aircraft.  Labora¬ 
tory  tests  had  demonstrated  that  the  new 
fluid  was  less  flammable  than  the  standard 
fluid.  However,  during  the  JLF  tests,  30 
percent  of  the  shots  on  the  new  fluid  re¬ 
sulted  in  fires  contrasted  with  15  percent  of 
the  shots  on  the  standard  fluid  (Reference 
198). 

While  much  insight  and  valuable  wisdom 
are  to  be  obtained  through  the  testing  of 
components  or  subsystems,  some  phenom¬ 
ena  are  only  observable  when  full-up  sys¬ 
tems  are  tested.  The  interaction  of  such 
phenomena  has  been  termed  "cascading 
damage."  Such  damage  is  a  result  of  the 
synergistic  damage  mechanisms  that  are  at 
work  in  the  "real  world"  and  likely  to  be 
found  during  actual  combat.  Live  Fire  Test¬ 


ing  provides  a  way  of  examining  the  dam¬ 
ages  inflicted  not  only  on  materiel  but  also 
on  personnel.  The  crew  casualty  problem  is 
an  important  issue  that  the  LFT  program  is 
addressing.  The  program  provides  an  op¬ 
portunity  to  assess  the  effects  of  the  com¬ 
plex  environments  that  crews  are  likely  to 
encounter  in  combat  (e.g.,  fire,  toxic  fumes, 
blunt  injury  shock  and  acoustic  injuries) 
(Reference  37). 

18.2.3  Use  of  Modeling 
and  Simulation 

Survivability  and  lethality  assessments 
have  traditionally  relied  largely  on  the  use 
of  modeling  and  simulation  techniques. 
The  Live  Fire  Test  Program  does  not  re¬ 
place  the  need  for  such  techniques;  in  fact, 
the  Live  Fire  Test  Guidelines  issued  by 
OSD  in  May  1987  (Figure  18-1)  required 
that  no  shots  be  conducted  until  pre-shot 
model  predictions  were  made  concerning 
the  expected  damage.  Such  predictions  are 
useful  for  several  reasons.  First,  they  assist 
in  the  test  planning  process.  If  a  model 
predicts  that  no  damage  will  be  inflicted, 
test  designers  and  planners  should  reex¬ 
amine  the  selection  of  the  shotlines  and/ or 
reassess  the  accuracy  of  the  threat  repre¬ 
sentation.  Second,  pre-shot  model  predic¬ 
tions  provide  the  Services  with  the  oppor¬ 
tunity  to  validate  the  accuracy  of  the  mod¬ 
els  by  comparing  them  with  actual  LFT 
results.  At  the  same  time,  the  LFT  program 
reveals  areas  of  damage  that  may  be  absent 
from  existing  models  and  simulations. 
Third,  pre-shot  model  predictions  can  be 
used  to  help  conserve  scarce  target  re¬ 
sources.  For  example,  models  can  be  used 
to  determine  a  sequence  of  shots  that  pro¬ 
vides  for  the  less-damaging  shots  to  be 
conducted  first,  followed  by  the  more-cata¬ 
strophic  shots  resulting  in  maximum  target 
damage. 
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Table  18-2.  Types  of  Live  Fire  Testing 
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Source:  “Live  Fire  Testing;  Evaluating  DoD’s  Program,” 

General  Accounting  Office,  GAO/PEMD-87-17,  August  1987. 
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Figure  18-1.  Live  Fire  T&E  Planning  Guide 
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18.2.4  Live  Fire  Test  Best  Practices 

The  DoD  5000.2-R  guidelines  state  that 
plans  for  live  fire  testing  must  be  included 
in  the  TEMP.  Key  points  covered  in  the  LFT 
guidelines  include  the  following: 

•  The  LFT&E  Detailed  T&E  Plan  is  the 
basic  planning  document  used  by  OSD  and 
the  Services  to  plan,  review  and  approve 
LFT&E.  Services  will  submitt  the  plan  to 
the  DOTE  for  comment  at  least  30  days 
prior  to  test  initiation. 

•  The  LFT&E  plan  must  contain  general 
information  on  the  system's  required  per¬ 
formance,  operational  and  technical  char¬ 
acteristics,  critical  test  objectives  and  the 
evaluation  process. 

•  Each  LFT&E  plan  must  include  testing 
of  complete  systems.  A  limited  set  of  live 
fire  tests  may  involve  production  compo¬ 
nents  configured  as  a  subsystem  before 
full-up  testing. 

•  A  Service  report  must  be  submitted 
within  120  days  of  the  completion  of  the 
live  fire  test.  The  report  must  include  the 
firing  results,  test  conditions,  limitations 
and  conclusions  and  be  submitted  in  classi¬ 
fied  and  unclassified  form. 

•  Within  45  days  of  receipt  of  the  Service 
report,  a  separate  Live  Fire  Test  Report 
(part  6,  DoD  5000.2-R)  will  be  produced  by 
the  DOTE,  approved  by  the  Secretary  of 
Defense,  and  transmitted  to  Congress.  The 
conclusions  of  the  report  will  be  indepen¬ 
dent  of  the  conclusions  of  the  Service  re¬ 
port.  Reporting  on  LFT&E  maybe  included 
in  the  weapon  system's  Beyond  Low  Rate 
Initial  Production  Report  completed  by  the 
DOTE. 

•  The  Congress  shall  have  access  to  all 
live  fire  test  data  and  all  live  fire  test 


reports  held  by  or  produced  by  the  Secre¬ 
tary  of  the  concerned  Service  or  by  OSD. 

•  The  costs  of  all  live  fire  tests  shall  be 
paid  from  funding  for  the  system  being 
tested.  In  some  instances,  the  Director, 
Test,  Systems  Engineering,  and  Evalua¬ 
tion  (DTSEE)  may  elect  to  supplement 
such  funds  for  the  acquisition  of  targets 
or  target  simulators,  although  the  ulti¬ 
mate  responsibility  rests  on  the  concerned 
Service. 

18.3  TESTING  FOR  NUCLEAR 
HARDNESS  AND  SURVIVABILITY 

18.3.1  Background 

Nuclear  survivability  must  be  incorpo¬ 
rated  into  the  design,  acquisition  and 
operation  of  all  systems  that  must  per¬ 
form  critical  missions  in  a  nuclear  envi¬ 
ronment.  Nuclear  survivability  is 
achieved  through  a  combination  of  four 
methods:  hardness,  avoidance,  prolifera¬ 
tion  and  reconstitution.  Hardness  allows 
a  system  to  physically  withstand  a  nuclear 
attack.  Avoidance  encompasses  measures 
taken  to  avoid  encountering  a  nuclear 
environment.  Proliferation  involves  hav¬ 
ing  sufficient  systems  to  compensate  for 
probable  losses.  Reconstitution  includes 
the  actions  taken  to  repair  or  resupply 
damaged  units  in  time  to  complete  a  mis¬ 
sion  satisfactorily. 

There  is  a  wide  variety  of  possible  effects 
from  a  nuclear  detonation.  They  include: 
electromagnetic  pulse  (EMP),  ionizing  ra¬ 
diation,  thermal  radiation,  blast,  shock, 
dust,  debris,  blackout  and  scintillation. 
Each  weapon  system  is  susceptible  to 
some  but  not  all  of  these  effects.  The  pro¬ 
gram  manager  (PM)  and  his/her  staff 
must  identify  the  effects  that  may  have  an 
impact  on  the  system  under  development 
and  manage  the  design,  development  and 
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testing  of  the  system  in  a  manner  that  mini¬ 
mizes  degradation.  The  variety  of  possible 
nuclear  effects  is  described  more  fully  in 
the  "Nuclear  Survivability  Handbook  for 
Air  Force  OT&E"  (Reference  91). 

18.3.2  Assessing  Nuclear  Survivability 
Throughout  The  System  Acquisition 
Cycle 

The  PM  must  ensure  that  nuclear  surviv¬ 
ability  issues  are  addressed  throughout  the 
system  acquisition  cycle.  During  the  Con¬ 
cept  Exploration  Phase,  the  survivability 
requirements  stated  in  the  Service  require¬ 
ments  document  should  be  verified,  re¬ 
fined  or  further  defined.  "Mission-critical 
systems  shall  be  survivable  to  the  threat 
levels  anticipated  in  their  operating  envi¬ 
ronment"  (part  4,  DoD  5000.2-R).  During 
the  Program  Definition  and  Risk  Reduc- 
tionPhase,  tradeoffs  between  hardness  lev¬ 
els  and  other  system  characteristics  (such 
as  weight,  decontaminability  and  compat¬ 
ibility)  should  be  described  quantitatively. 
Tradeoffs,  between  hardness,  avoidance, 
proliferation  and  reconstitution  as  a  method 
for  achieving  survivability,  should  also  be 
considered  at  this  time.  During  the  Engi¬ 
neering  and  Manufacturing  Development 
Phase,  the  system  must  be  adequately  tested 
to  confirm  that  hardness  objectives,  crite¬ 
ria,  requirements  and  specifications  are  met. 
Plans  for  nuclear  hardness  and  survivabil¬ 
ity  testing  should  be  addressed  in  the  TEMP . 
The  appropriate  commands  must  make 
provision  for  test  and  hardness  surveil¬ 
lance  equipment  and  procedures  so  re¬ 
quired  hardness  levels  can  be  maintained 
once  the  system  is  operational. 

During  the  Production ,  Fielding/Deploy- 
ment  and  Operational  Support  Phase,  sys¬ 
tem  hardness  is  maintained  through  an 
active  hardness  assurance  program.  Such  a 
program  ensures  that  the  end  product  con¬ 


forms  to  hardness  design  specifications 
and  that  hardness  aspects  are  reevalu¬ 
ated  before  any  retrofit  changes  are  made 
to  existing  systems. 

Once  a  system  is  operational,  a  hardness 
surveillance  program  may  be  implemented 
to  maintain  system  hardness  and  to  iden¬ 
tify  any  further  evaluation,  testing  or  retro¬ 
fit  changes  required  to  ensure  survivabil¬ 
ity.  A  hardness  surveillance  program  con¬ 
sists  of  a  set  of  scheduled  tests  and  inspec¬ 
tions  to  ensure  that  a  system's  designed 
hardness  is  not  degraded  through  opera¬ 
tional  use,  logistic  support,  maintenance 
actions  or  natural  causes. 

18.3.3  Test  Planning 

The  "Nuclear  Survivability  Handbook  for 
Air  Force  OT&E"  describes  the  following 
challenges  associated  with  nuclear  hard¬ 
ness  and  survivability  testing: 

(1)  The  magnitude  and  range  of  effects 
from  a  nuclear  burst  are  much  greater  than 
those  from  conventional  explosions  that 
may  be  used  to  simulate  nuclear  bursts. 
Nuclear  detonations  have  effects  not  found 
in  conventional  explosions.  The  intense 
nuclear  radiation,  blast,  shock,  thermal  and 
EMP  fields  are  difficult  to  simulate.  In  addi¬ 
tion,  systems  are  often  tested  at  stress  levels 
that  are  either  lower  than  those  established 
by  the  criteria  or  lower  than  the  level  needed 
to  cause  damage  to  the  system. 

(2)  The  yields  and  configurations  for  un¬ 
derground  testing  are  limited.  It  is  gener¬ 
ally  not  possible  to  test  all  relevant  effects 
simultaneously  or  to  observe  possibly  im¬ 
portant  synergism  between  effects. 

(3)  System-level  testing  for  nuclear  ef¬ 
fects  is  normally  expensive,  takes  years  to 
plan  and  conduct  and  requires  specialized 
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expertise.  Often,  classes  of  tests  conducted 
early  in  the  program  are  not  repeated  later. 
Therefore,  operational  requirements  should 
be  folded  into  these  tests  from  the  start, 
often  early  in  the  acquisition  process.  This 
mandates  a  more-extensive,  combined 
DT&E/OT&E  (operational  test  and  evalu¬ 
ation)  test  program  than  normally  found  in 
other  types  of  testing. 

Program  managers  and  test  managers  must 
remain  sensitive  to  the  ambiguities  involved 
in  testing  for  nuclear  survivability.  For  ex¬ 
ample,  there  is  no  universal  quantitative 
measure  of  survivability;  and  statements  of 
survivability  may  lend  themselves  to  a  va¬ 
riety  of  interpretations.  Moreover,  it  can  be 
difficult  to  combine  system  vulnerability 
estimates  for  various  nuclear  effects  into  an 
assessment  of  overall  survivability.  As  a 
result,  program/ test  managers  must  exer¬ 
cise  caution  when  developing  test  objec¬ 
tives  and  specifying  measures  of  merit  re¬ 
lated  to  nuclear  survivability. 

18.3.4  Test  Execution 

For  nuclear  hardness  and  survivability  test¬ 
ing,  development  test  (DT)  and  operational 
test  (OT)  efforts  are  often  combined  be¬ 
cause  it  is  not  possible  to  test  in  an  opera¬ 
tional  nuclear  environment.  The  use  of  an 
integrated  DT/OT  program  requires  early 
and  continuous  dialogue  between  the  two 
test  communities  so  each  understands  the 
needs  of  the  other  and  maximum  coopera¬ 
tion  in  meeting  objectives  is  obtained. 


Test  and  evaluation  techniques  available  to 
validate  the  nuclear  survivability  aspects 
of  systems  and  subsystems  include  under¬ 
ground  nuclear  testing,  environmental 
simulation  (system  level,  subsystem  level 
and  component  level)  and  analytical  simu¬ 
lation.  Table  18-3  outlines  the  major  activi¬ 
ties  relevant  to  the  assessment  of  nuclear 
hardness  and  survivability  and  the  phases 
of  the  acquisition  cycle  in  which  they  occur. 

18.4  SUMMARY 

The  survivability  and  lethality  aspects  of 
certain  system  must  be  evaluated  through 
live  fire  tests.  These  tests  are  used  to  pro¬ 
vide  insights  into  the  weapon  system's  abil¬ 
ity  to  continue  to  operate/ fight  after  being 
hit  by  enemy  threat  systems.  It  provides  a 
way  of  examining  the  damages  inflicted 
not  only  on  materiel  but  also  on  personnel. 
Live  fire  testing  also  provides  an  opportu¬ 
nity  to  assess  the  effects  of  complex  envi¬ 
ronments  that  crews  are  likely  to  encounter 
in  combat. 

Nuclear  survivability  must  be  carefully 
evaluated  during  the  system  acquisition 
cycle.  Tradeoffs  between  hardness  levels 
and  other  system  characteristics,  such  as 
weight,  speed,  range,  cost,  etc.,  must  be 
evaluated.  Nuclear  survivability  testing  is 
difficult,  and  the  evaluation  of  test  results 
may  lend  itself  to  a  variety  of  interpreta¬ 
tions.  Therefore,  PMs  must  exercise  cau¬ 
tion  when  developing  test  objectives  re¬ 
lated  to  nuclear  survivability. 
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Table  18-3.  Nuclear  Hardness  and  Survivability  Assessment  Activities 

CONCEPT  EXPLORATION  PHASE 

•  PREPARATION  OF  TEST  AND  EVALUATION  MASTER  PLAN  (TEMP)  THAT  INCLUDES  INITIAL  PLANS 
FOR  NUCLEAR  HARDNESS  AND  SURVIVABILITY  (NH&S)  TESTS 

-  IDENTIFICATION  OF  NH&S  REQUIREMENTS  IN  VERIFIABLE  TERMS 

-  IDENTIFICATION  OF  SPECIAL  NH&S  TEST  FACILITY  REQUIREMENTS  WITH  EMPHASIS  ON 
LONG  LEAD  TIME  ITEMS 

•  DEVELOPMENT  OF  NUCLEAR  CRITERIA 

PROGRAM  DEFINITION  AND  RISK  REDUCTION  PHASE 

•  INCREASE  TEST  PLANNING 

•  TEMP  UPDATE 

.  CONDUCT  OF  NH&S  TRADE  STUDIES 

-  NH&S  REQUIREMENTS  VERSUS  OTHER  SYSTEM  REQUIREMENTS 

-  ALTERNATE  METHODS  FOR  ACHIEVING  NH&S 

•  CONDUCT  OF  LIMITED  TESTING 

-  PIECE-PART  HARDNESS  TESTING 

-  DESIGN  CONCEPT  TRADE-OFF  TESTING 

-  TECHNOLOGY  DEMONSTRATION  TESTING 

«  DEVELOPMENT  OF  PERFORMANCE  SPECIFICATIONS  THAT  INCLUDE  QUANTITATIVE  HARDNESS 
LEVELS 

ENGINEERING  AND  MANUFACTURING  DEVELOPMENT  PHASE 

•  FIRST  OPPORTUNITY  TO  TEST  PROTOTYPE  HARDWARE 

•  TEMP  UPDATE 

•  DEVELOPMENT  OF  NUCLEAR  HARDNESS  DESIGN  HANDBOOK 

-  PRIOR  TO  PRELIMINARY  DESIGN  REVIEW 

-  USUALLY  PREPARED  BY  NUCLEAR  EFFECTS  SPECIALTY  CONTRACTOR 

•  CONDUCT  OF  TESTING 

-  PRE-CRITICAL  DESIGN  REVIEW  (CDR)  DEVELOPMENT  AND  QUALIFICATION  TESTS 

-  DEVELOPMENTAL  TESTING  ON  NUCLEAR-HARDENED  PIECE  PARTS,  MATERIALS,  CABLING, 

AND  CIRCUITS 

-  NH&S  BOX  AND  SUBSYSTEM  QUALIFICATION  TESTS  (POST-CDR) 

-  ACCEPTANCE  TESTS  TO  VERIFY  HARDWARE  MEETS  SPECIFICATIONS  (POST-CDR,  PRIOR 
TO  FIRST  DELIVERY) 

-  SYSTEM-LEVEL  HARDNESS  ANALYSIS  (USING  BOX  AND  SUBSYSTEM  TEST  RESULTS) 

-  SYSTEM-LEVEL  NH&S  TEST 

PRODUCTION,  FIELDING/DEPLOYMENT  AND  OPERATIONAL  SUPPORT  PHASE 

•  IMPLEMENTATION  OF  PROGRAM  TO  ENSURE  SYSTEM  RETAINS  ITS  NH&S  PROPERTIES 

•  SCREENING  OF  PRODUCTION  HARDWARE  FOR  HARDNESS 

•  IMPLEMENTATION  BY  USER  OF  PROCEDURES  TO  ENSURE  SYSTEM'S  OPERATION,  LOGISTIC 
SUPPORT  AND  MAINTENANCE  DO  NOT  DEGRADE  HARDNESS  FEATURES 

•  REASSESSMENT  OF  SURVIVABILITY  THROUGHOUT  SYSTEM  LIFE  CYCLE 
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LOGISTICS  INFRASTRUCTURE 

T&E 


19.1  INTRODUCTION 

In  all  materiel  acquisition  programs,  the 
logistics  support  effort  begins  in  the  Mis¬ 
sion  Area  Analysis  Phase  before  program 
initiation,  continues  throughout  the  acqui¬ 
sition  cycle  and  extends  past  the  deploy¬ 
ment  phase.  Logistics  support  system  test¬ 
ing  must,  therefore,  extend  over  the  entire 
acquisition  cycle  of  the  system  and  be  care¬ 
fully  planned  and  executed  to  ensure  the 
readiness  and  supportability  of  the  system. 
This  chapter  covers  the  development  of 
logistics  support  test  requirements  and  the 
conduct  of  supportability  assessments  to 
ensure  that  readiness  and  supportability 
objectives  are  identified  and  achieved.  The 
importance  of  the  logistics  manager's  par¬ 
ticipation  in  the  Test  and  Evaluation  Mas¬ 
ter  Plan  (TEMP)  development  process 
should  be  stressed.  The  logistics  manager 
must  ensure  the  logistics  system  test  and 
evaluation  (T&E)  objectives  are  considered 
and  that  adequate  resources  are  available 
for  logistics  support  system  T&E. 

Logistic  system  support  planning  is  a  disci¬ 
plined,  unified  and  iterative  approach  to 
the  management  and  technical  activities 
necessary  to  integrate  support  consider¬ 
ations  into  system  and  equipment  design; 
develop  support  requirements  that  are  re¬ 
lated  consistently  to  readiness  objectives, 
design  and  each  other;  acquire  the  required 
support;  and  provide  the  required  support 
during  post-Milestone  (MS)  III  Operational 
Support  at  affordable  cost. 


Logistic  support  systems  are  usually  cat¬ 
egorized  into  10  specific  components,  or 
elements: 

(1)  Maintenance  planning 

(2)  Manpower  and  personnel 

(3)  Supply  support 

(4)  Support  equipment 

(5)  Technical  data 

(6)  Training  and  training  support 

(7)  Computer  resources  support 

(8)  Facilities 

(9)  Packaging,  handling,  storage  and 
transportation 

(10)  Design  interface. 

19.2  PLANNING  FOR  LOGISTIC 
SUPPORT  SYSTEM  T&E 

19.2.1  Objectives  of  Logistic 
System  T&E 

The  main  objective  of  logistic  system  T&E 
is  to  verify  that  the  logistic  support  being 
developed  for  the  materiel  system  is  ca¬ 
pable  of  meeting  the  required  objectives  for 


peacetime  and  wartime  employment.  This 
T&E  consists  of  the  usual  development  test 
and  evaluation  (DT&E)  and  operational 
test  and  evaluation  (OT&E)  but  also  in¬ 
cludes  post  deployment  supportability  as¬ 
sessments.  The  formal  DT&E  and  OT&E 
begin  in  the  Concept  Exploration  Phase 
and  continue  throughout  the  Production, 
Fielding /Deployment  and  Operational 
Support  Phase.  Figure  19-1,  drawn  from 
the  Defense  Systems  Management  College’s 
Integrated  Logistics  Support  Guide,  describes 
the  specific  development  test  (DT),  opera¬ 
tional  test  (OT)  and  supportability  assess¬ 
ment  objectives  for  each  acquisition  phase. 

19.2.2  Planning  for  Logistic 
Support  System  T&E 

19.2.2.1  Logistic  Support 
Planning 

The  logistic  support  manager  for  a  mate¬ 
riel  acquisition  system  is  responsible  for 
developing  the  logistic  support  planning 
which  documents  planning  for  and  imple¬ 
menting  the  support  of  the  fielded  sys¬ 
tem.  It  is  initially  prepared  during  the 
Concept  Exploration  Phase,  and  progres¬ 
sively  developed  in  more  detail  as  the 
system  moves  through  the  acquisition 
phases.  Identification  of  the  specific  lo¬ 
gistic  support  test  issues  related  to  the 
individual  logistics  elements  and  the  over¬ 
all  system  support  and  readiness  objec¬ 
tives  are  included. 

The  logistic  support  manager  is  assisted 
throughout  the  system's  development  by 
a  Logistics  Support  Integrated  Product 
Team  (IPT)  which  is  formed  early  in  the 
acquisition  cycle.  The  LS  IPT  is  a  coordi¬ 
nation/advisory  group  composed  of  per¬ 
sonnel  from  the  program  management 
office,  the  using  command  and  other  com¬ 
mands  concerned  with  acquisition  activi¬ 
ties  such  as  logistics,  testing  and  training. 


19.2.2.2  Supportability 
Assessment  Planning 

Based  upon  suitability  objectives,  the  logis¬ 
tic  support  manager,  in  conjunction  with 
the  system's  test  manager,  develops  suit¬ 
ability  assessment  planning.  This  planning 
identifies  the  testing  approach  and  the 
evaluation  criteria  that  will  be  used  to  as¬ 
sess  the  supportability-related  design  re¬ 
quirements  (e.g.,  reliability  and  maintain¬ 
ability)  and  adequacy  of  the  planned  logis¬ 
tic  support  resources  for  the  materiel  sys¬ 
tem.  Development  of  the  suitability  T&E 
planning  begins  in  the  Concept  Explora¬ 
tion  Phase;  the  planning  is  then  updated 
and  refined  in  each  successive  acquisition 
phase.  The  logistic  support  manager  may 
apply  the  best  practices  of  logistic  support 
analysis  as  described  in  MIL-HDBK-1388- 
1  A.  Test  and  evaluation  strategy  is  formu¬ 
lated,  T&E  program  objectives  and  criteria 
are  established  and  required  test  resources 
are  identified.  The  logistic  support  man¬ 
ager  ensures  that  T&E  strategy  is  based 
upon  quantified  supportability  require¬ 
ments  and  addresses  supportability  issues 
including  those  with  a  high  degree  of  asso¬ 
ciated  risk.  Also,  the  logistic  support  man¬ 
ager  ensures  that  the  necessary  quantities 
and  types  of  data  will  be  collected  during 
system  development  and  after  deployment 
of  the  system  to  validate  the  various  T&E 
objectives.  The  T&E  objectives  and  criteria 
must  provide  a  basis  that  ensures  critical 
supportability  issues  and  requirements  are 
resolved  or  achieved  within  acceptable  con¬ 
fidence  levels. 

19.2.2.3  Test  and  Evaluation 
Master  Plan  (TEMP) 

The  program  manager  must  include  suit¬ 
ability  T&E  information  in  the  TEMP  as 
specified  in  Department  of  Defense  (DoD) 
5000. 2-R.  The  input,  derived  from  logistic 
supportability  planning  with  the  assistance 
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of  the  logistic  support  manager  and  the 
tester,  includes  descriptions  of  required 
operational  suitability,  specific  plans  for 
testing  logistics  supportability,  and  re¬ 
quired  testing  resources.  It  is  of  critical 
importance  that  all  key  test  resources  re¬ 
quired  for  ILS  testing  (DT,  OT,  and  post 
deplo)mient  supportability)  be  identified 
in  the  TEMP  because  the  TEMP  provides  a 
long-range  alert  upon  which  test  resources 
are  budgeted  and  obtained  for  testing. 

19.2.3  Planning  Guidelines 
for  Logistic  Support  System  T&E 

The  following  guidelines  were  selected 
from  those  listed  in  the  Defense  Systems 
Management  College's  Integrated  Logistic 
Support  Guide: 

(1)  Develop  a  test  strategy  for  each  logis¬ 
tics  support-related  objective.  Ensure  that 
OT&E  planning  encompasses  all  logistic 
support  elements.  The  general  objectives 
shown  in  Figure  19-1  must  be  translated 
into  detailed  quantitative  and  qualitative 
requirements  for  each  acquisition  phase 
and  each  T&E  program. 

(2)  Incorporate  logistic  support  testing 
requirements  (where  feasible)  into  the  for¬ 
mal  DT&E/OT&E  plans. 

(3)  Identify  logistic  support  T&E  that 
will  be  performed  outside  of  the  normal 
DT&E  and  OT&E.  Include  subsystems  that 
require  off-system  evaluation. 

(4)  Identify  all  required  resources,  in¬ 
cluding  test  articles  and  logistic  support 
items  for  formal  DT/OT  and  separate  lo¬ 
gistic  support  system  testing  (participate 
with  test  planner). 

(5)  Ensure  establishment  of  an  opera¬ 
tionally  realistic  test  environment,  to  in¬ 
clude  personnel  representatives  of  those 


who  will  eventually  operate  and  maintain 
the  fielded  system.  These  personnel  should 
be  trained  for  the  test  using  prototypes  of 
the  actual  training  courses  and  devices. 
They  should  be  supplied  with  drafts  of  all 
technical  manuals  and  documentation  that 
will  be  used  with  the  fielded  system. 

(6)  Ensure  planned  OT&E  will  provide 
sufficient  data  on  high-cost  and  high-main¬ 
tenance  burden  items  (e.g.,  for  high-cost 
critical  spares,  early  test  results  can  be  used 
to  reevaluate  selection). 

(7)  Participate  early  and  effectively  in  the 
TEMP  development  process  to  ensure  the 
TEMP  includes  critical  logistic  T&E  desig¬ 
nated  test  funds  from  program  and  budget 
documents. 

(8)  Identify  the  planned  utilization  of  all 
data  collected  during  the  assessments  to 
avoid  mismatching  of  data  collection  and 
information  requirements. 

Detailed  evaluation  criteria  for  each  of  the 
10  logistic  support  elements  have  been  pre¬ 
sented  in  Department  of  the  Army  Pam¬ 
phlet  700-50,  "Integrated  Logistic  Support: 
Developmental  Supportability  Test  and 
Evaluation  Guide." 

Additional  guidance  may  be  found  in  the 
Logistics  Test  and  Evaluation  Handbook  de¬ 
veloped  by  the  412  Logistics  Group, 
Edwards  AFB,  CA. 

19.3  CONDUCTING  LOGISTIC 
SUPPORT  SYSTEM  T&E 

19.3.1  The  Process 

The  purposes  of  logistic  support  system 
T&E  are  to  measure  the  supportability  of  a 
developing  system  throughout  the  acquisi¬ 
tion  process,  to  identify  supportability  defi¬ 
ciencies  and  potential  corrections/improve- 
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Figure  19-1 .  Logistics  Supportabiiity  Objectives  in  the  T&E  Program 


merits  as  test  data  becomes  available,  and 
to  assess  the  operational  suitability  of  the 
planned  support  system.  It  also  evaluates 
the  system's  ability  to  achieve  planned 
readiness  objectives  for  the  system/equip¬ 
ment  being  developed .  Specific  logistic  sup¬ 
port  system  T&E  tasks  (guidance  in  MIL- 
HDBK-1388-1A)  include: 

•  Analysis  of  test  results  to  verify  achieve¬ 
ment  of  specified  supportability  require¬ 
ments; 

•  Determination  of  improvements  in  sup- 
portability  and  supportability-related  de¬ 
sign  parameters  needed  for  the  system  to 
meet  established  goals  and  thresholds; 

•  Identification  of  areas  where  estab¬ 
lished  goals  and  thresholds  have  not  been 
demonstrated  within  acceptable  confidence 
levels; 

•  Development  of  corrections  for  identi¬ 
fied  supportability  problems  such  as  modi¬ 
fications  to  hardware,  software,  support 
plans,  logistic  support  resources  or  opera¬ 
tional  tactics; 

•  Projection  of  changes  in  costs, 
readiness  and  logistic  support  re¬ 
sources  due  to  implementation  of  cor¬ 
rections; 

•  Analysis  of  supportability  data 
from  the  deployed  system  to  verify 
achievement  of  the  established  goals 
and  thresholds  and  where  operational 
results  deviate  from  projections,  de¬ 
termination  of  the  causes  and  correc¬ 
tive  actions. 

Logistics  support  system  T&E  may  consist 
of  a  series  of  logistics  demonstrations  and 
assessments  that  are  usually  conducted  as 
part  of  system  performance  tests.  Special 
end-item  equipment  tests  are  rarely  con¬ 
ducted  solely  for  logistic  parameter  evalu¬ 
ation. 


19.3.2  Reliability  and  Maintainability 

System  availability  is  generally  considered 
to  be  composed  of  two  major  system  char¬ 
acteristics  —  reliability  and  maintainabil¬ 
ity.  The  DoD  5000.2-R  states: 

Reliability,  maintainability,  and  avail¬ 
ability  requirements  shall  be  based  on 
operational  requirements  and  life-cycle 
cost  considerations;  stated  in  quantifi¬ 
able,  operational  terms;  measurable 
during  developmental  and  operational 
test  and  evaluation;  and  defined  for  all 
elements  of  the  system,  including  sup¬ 
port  and  training  equipment. 

Reliability  (R)  is  the  ability  of  a  system  and 
its  parts  to  perform  its  mission  without 
failure,  degradation,  or  demand  on  the  sup¬ 
port  system. 

Maintainability  (M)  is  the  ability  of  an  item 
to  be  retained  in  or  restored  to  specified 
condition  when  maintenance  is  performed 
by  personnel  having  specific  skill  levels, 
using  prescribed  procedures  and  resources, 
at  each  prescribed  level  of  maintenance 
and  repair. 

Operational  Reliability  and  Maintainabil¬ 
ity  Value  is  any  measure  of  reliability  or 
maintainability  that  includes  the  combined 
effects  of  item  design,  quality,  installation, 
environment,  operation,  maintenance,  and 
repair. 

The  R  and  M  program  objectives  are  to  be 
defined  as  system  parameters  early  in  the 
development  process.  They  will  be  used  as 
evaluation  criteria  throughout  the  design, 
development  and  production  processes. 
Reliability  and  maintainability  objectives 
should  be  translated  into  quantifiable  con¬ 
tractual  terms  and  allocated  through  the 
system  design  hierarchy.  An  understanding 
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of  how  this  allocation  affects  testing  oper¬ 
ating  characteristics  below  system  level  can 
be  found  in  DoD  3235.1-H,  "T&E  of  System 
Reliability,  Availability  and  Maintainabil¬ 
ity."  This  is  especially  important  to  testing 
organizations  expected  to  make  early  pre¬ 
dictions  of  system  performance.  Guidance 
on  testing  reliability  may  also  be  found  in 
MIL-HDBK-78I,  "Reliability  Testing  for  En¬ 
gineering  Development,  Qualification,  and 
Production." 

19.3.2.1  Reliability 

Guidelines  for  reliability  evaluation  are  to 
be  published  in  a  non-government  stan¬ 
dard  to  replace  MIL-STD-785: 

Environmental  Stress  Screening  (ESS)  is  a 
test,  or  series  of  tests  during  engineering 
development,  specifically  designed  to  iden¬ 
tify  weak  parts  or  manufacturing  defects. 
Test  conditions  should  stimulate  failures 
typical  of  early  field  service  rather  than 
provide  an  operational  life  profile. 

Reliability  Development/ Growth  Testing 
(RDT/RGT)  is  a  systematic  engineering 
process  of  test-analyze-fix-retest  (TAFT) 
where  equipment  is  tested  under  actual, 
simulated,  or  accelerated  environments.  It 
is  an  iterative  methodology  intended  to 
rapidly  and  steadily  improve  reliability. 
Test  articles  are  usually  subjected  to  ESS 
prior  to  beginning  RDT/RGT  to  eliminate 
those  with  manufacturing  defects. 

Reliability  Qualification  Test  (RQT)  is  to 
verify  that  threshold  reliability  require¬ 
ments  have  been  met  before  items  are  com¬ 
mitted  to  production.  A  statistical  test  plan 
is  used  to  predefine  criteria  which  will  limit 
government  risk.  Test  conditions  must  be 
operationally  realistic. 

Production  Reliability  Acceptance  Test 
(PRAT)  is  intended  to  simulate  in-service 


use  of  the  delivered  item  or  production  lot. 
"Because  it  must  provide  a  basis  for  deter¬ 
mining  contractual  compliance,  and  be¬ 
cause  it  applies  to  the  items  actually  deliv¬ 
ered  to  operational  forces,  PRAT  must  be 
independent  of  the  supplier  if  at  all  pos¬ 
sible."  PRAT  may  require  expensive  test 
facilities,  so  100  percent  sampling  is  not 
recommended. 


19.3.2.2  Maintainability 

Maintainability  design  factors  and  test/dem¬ 
onstration  requirements  used  to  evaluate 
maintainability  characteristics  must  be  based 
on  program  objectives  and  thresholds.  Areas 
for  evaluation  might  include  (DoD  3235.1-H): 

Accessibility:  Assess  how  easily  the  item 
can  be  repaired  or  adjusted. 

Visibility:  Assess  the  ability /need  to  see 
the  item  being  repaired. 

Testability:  Assess  ability  to  detect  and 
isolate  system  faults  to  the  faulty  replace¬ 
able  assembly  level. 

Complexity:  Assess  the  impact  of  the 
number,  locationand  characteristic  (stan¬ 
dard  or  special  purpose)onsystem  main¬ 
tenance. 

Interchangeability:  Assess  the  level  of 
difficulty  encountered  when  failed  or 
malfunctioning  parts  are  removed  or  re¬ 
placed  with  an  identical  part  not  requir¬ 
ing  recalibration. 

A  true  assessment  of  system  maintainability 
generally  must  be  developed  at  the  system 
level  under  operating  conditions  and  using 
production  configuration  hardware.  There¬ 
fore  a  maintainability  demonstration 
(guidelines  in  MIL-HDBK-470)  should  be 
conducted  prior  to  Milestone  III. 
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19.3.3  T&E  of  System  Support 
Package 

The  T&E  of  the  support  for  a  materiel 
system  requires  a  system  support  pack¬ 
age  consisting  of  spares,  support  equip¬ 
ment,  technical  documents  and  publica¬ 
tions,  representative  personnel,  any  pe¬ 
culiar  support  requirements  and  the  test 
article  itself,  in  short,  all  of  the  items  that 
would  eventually  be  required  when  the 
system  is  operational.  This  complete  sup¬ 
port  package  must  be  at  the  test  site  be¬ 
fore  the  test  is  scheduled  to  begin.  Delays 
in  the  availability  of  certain  support  items 
could  prevent  the  test  from  proceeding 
on  schedule.  This  could  be  costly  due  to 
on-site  support  personnel  on  hold  or 
tightly  scheduled  system  ranges  and  ex¬ 
pensive  test  resources  not  being  properly 
utilized.  Also,  it  could  result  in  the  test 
proceeding  without  conducting  the  com¬ 
plete  evaluation  of  the  support  system. 
The  logistic  support  test  planner  must 
ensure  that  the  required  personnel  are 
trained  and  available,  the  test  facility 
scheduling  is  flexible  enough  to  permit 
normal  delays,  and  the  test  support  pack¬ 
age  is  on  site  on  time. 

19.3.4  Data  Collection 
and  Analysis 

The  logistic  support  manager  must  coordi¬ 
nate  with  the  testers  to  ensure  that  the 
methods  used  for  collection,  storage  and 
extraction  of  logistic  support  system  T&E 
data  are  compatible  with  those  used  in 
testing  the  materiel  system.  As  with  any 
testing,  the  test  planning  must  ensure  that 
all  required  data  is  identified;  it  is  sufficient 
to  evaluate  a  system's  readiness  and  sup- 
portability;  and  plans  are  made  for  a  data 
management  system  that  is  capable  of  the 
data  classification,  storage,  retrieval  and 
reduction  necessary  for  statistical  analysis. 
Large  statistical  sample  sizes  may  require  a 


common  database  that  integrates  contrac¬ 
tor,  DT&E  and  OT&E  data  so  required  per¬ 
formance  parameters  can  be  demonstrated . 

19.3.5  Use  of  Logistic  Support  System 
Test  Results 

The  emphasis  on  the  use  of  the  results  of 
testing  changes  as  the  program  moves  from 
the  Concept  Exploration  Phase  to  post  de¬ 
ployment.  During  early  phases  of  a  pro¬ 
gram,  the  evaluation  results  are  used  pri¬ 
marily  to  verify  analysis  and  develop  fu¬ 
ture  projections.  As  the  program  moves 
into  the  Engineering  and  Manufactuing 
Development  Phase  and  hardware  becomes 
available,  the  evaluation  addresses  design, 
particularly  the  reliability  and  maintain¬ 
ability  aspects;  training  programs;  support 
equipment  adequacy;  personnel  skills  and 
availability;  and  technical  publications. 

The  logistic  support  system  manager  must 
make  the  program  manager  (PM)  aware  of 
the  impact  on  the  program  of  logistical 
shortcomings  that  are  identified  during  the 
T&E  process.  The  PM,  in  turn,  must  ensure 
that  the  solutions  to  any  shortcomings  are 
identified  and  reflected  in  the  revised  speci¬ 
fications  and  that  the  revised  test  require¬ 
ments  are  included  in  the  updated  TEMP 
as  the  program  proceeds  through  the  vari¬ 
ous  acquisition  stages. 

19.4  LIMITATIONS  TO  LOGISTIC 
SUPPORT  SYSTEM  T&E 

Concurrent  testing  or  tests  that  have  accel¬ 
erated  schedules  frequently  do  not  have 
sufficient  test  articles,  equipment  or  hard¬ 
ware  to  achieve  statistical  confidence  in  the 
testing  conducted .  DoD 5000. 2-R  stipulates 
that  support  resources  such  as  operator 
and  maintenance  manuals,  tools,  support 
equipment,  training  devices,  etc.  for  major 
weapon  system  components  shall  not  be 
procured  before  the  weapons  system/ com- 
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ponent  hardware  and  software  design  sta¬ 
bilizes.  The  shortage  of  equipment  is  often 
the  reason  that  shelf-life  and  service-life 
testing  is  incomplete,  leaving  the  logistic 
support  system  evaluator  with  insufficient 
data  to  predict  future  performance  of  the 
test  item.  Some  evaluations  must  measure 
performance  against  a  point  on  the 
parameter's  growth  curve.  The  logistic  sup¬ 
port  system  testing  will  continue  post  pro¬ 
duction  to  obtain  required  sample  sizes  for 
verifying  performance  criteria.  Many  as¬ 
pects  of  the  logistic  support  system  may 
not  be  available  for  initial  operational  test 
and  evaluation  (lOT&E)  and  become  test¬ 
ing  limitations.  The  PMO  must  develop 
enough  logistic  support  to  ensure  the  user 
can  maintain  the  system  during  lOT&E 
without  requiring  system  contractor  in¬ 
volvement  (legislated  constraints).  Any  lo¬ 
gistics  support  system  limitations  upon 
lOT&E  will  likely  be  evaluated  during  fol¬ 
low  on  operational  test  and  evaluation. 


20.5  SUMMARY 

Test  and  evaluation  are  the  logisticians' 
tools  for  measuring  the  ability  of  the 
planned  support  system  to  fulfill  the  mate¬ 
riel  system's  readiness  and  supportability 
objectives.  The  effectiveness  of  logistic  sup¬ 
port  system  T&E  is  based  upon  the  com¬ 
pleteness  and  timeliness  of  the  planning 
effort. 

The  logistics  support  system  T&E  require¬ 
ments  must  bean  integral  part  of  the  TEMP 
to  ensure  budgeting  and  scheduling  of  re¬ 
quired  test  resources.  Data  requirements 
must  be  completely  identified,  with  ad¬ 
equate  plans  made  for  collection,  storage, 
retrieval  and  reduction  of  test  data.  At  MS 
in,  decision  makers  can  expect  that  some 
logistics  system  performance  parameters 
will  not  have  finished  testing  because  of  the 
large  sample  sizes  required  for  statistical 
analysis. 
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EC/CISR  TEST  AND  EVALUATION 


20.1  INTRODUCTION 

Testing  of  electronic  combat  (EC)  and  com¬ 
mand,  control,  communications,  comput¬ 
ers,  surveillance,  and  reconnaissance 
(C^ISR)  systems  pose  unique  problems  for 
the  tester  because  of  the  difficulty  in  mea¬ 
suring  their  operational  performance.  Com¬ 
patibility,  interoperability  and  integration 
are  key  performance  areas  for  these  sys¬ 
tems.  Special  testing  techniques  and  facili¬ 
ties  are  normally  required  in  EC  and  C^ISR 
testing.  This  chapter  discusses  the  prob¬ 
lems  associated  with  EC  and  C^ISR  test¬ 
ing  and  presents  methodologies  the  tester 
can  consider  using  to  overcome  the  prob¬ 
lems. 

20.2  TESTING  EC  SYSTEMS 

20.2.1  Special  Consideration 
When  Testing  EC  Systems 

Electronic  combat  systems  operate  across 
the  electromagnetic  spectrum,  performing 
offensive  and  defensive  support  roles.  Con¬ 
figurations  vary  from  subsystem  compo¬ 
nents  to  full-up  independent  systems.  The 
EC  systems  are  used  to  increase  survivabil¬ 
ity,  degrade  enemy  capability  and  contrib¬ 
ute  to  the  overall  success  of  the  combat 
mission.  Decision  makers  want  to  know 
the  incremental  contribution  to  total  force 
effectiveness  made  by  a  new  EC  system 
when  measured  in  a  force-on-force  engage¬ 
ment.  However,  the  contractual  specifica¬ 
tions  for  EC  systems  are  usually  stated  in 
terms  of  engineering  parameters  such  as 


effective  radiated  power,  reduction  in  com¬ 
munications  intelligibility  and  jamming- 
to-signal  ratio.  These  measures  are  of  little 
use  by  themselves  in  assessing  contribu¬ 
tion  to  mission  success.  The  decision  mak¬ 
ers  require  that  testing  be  conducted  under 
realistic  operational  conditions;  but  the 
major  field  test  ranges,  such  as  the  shore¬ 
line  at  Eglin  AFB  or  the  desert  at  Nellis 
AFB,  cannot  provide  the  signal  density  or 
realism  of  threats  that  would  be  presented 
by  regional  conflicts  in  central  Europe.  In 
field  testing,  the  tester  can  achieve  one-on- 
one  or,  at  best,  few-on-few  testing  condi¬ 
tions.  To  do  this  the  tester  needs  a  method¬ 
ology  that  will  permit  extrapolation  of  en¬ 
gineering  measurements  and  one-on-one 
test  events  to  create  more  operationally 
meaningful  measures  of  mission  success  in 
a  force-on-force  context,  usually  under 
simulated  conditions. 

20.2.2  Integrated  Test  Approach 

An  integrated  approach  to  EC  testing  using 
a  combination  of  large-scale  models,  com¬ 
puter  simulations,  hybrid  man-in-the-loop 
simulators  and  field  test  ranges  is  a  solu¬ 
tion  for  the  EC  tester.  No  tool  by  itself  is 
adequate  to  provide  a  comprehensive 
evaluation.  Simulation,  both  digital  and 
hybrid,  can  provide  a  means  for  efficient 
test  execution.  Computer  models  can  be 
used  to  simulate  many  different  test  cases 
to  aid  the  tester  in  assessing  the  critical 
test  issues  (i.e.,  sensitivity  analysis)  and 


produce  a  comprehensive  set  of  predicted 
results.  As  digital  simulation  models  are 
validated  with  empirical  data  from  testing, 
they  can  be  used  to  evaluate  the  system 
under  test  in  a  more  dense  and  complex 
threat  environment  and  at  expected  war¬ 
time  levels.  In  addition,  the  field  test  results 
are  used  to  validate  the  model;  and  the 
model  is  used  to  validate  the  field  tests, 
thus  lending  more  credibility  to  both  re¬ 
sults.  Hybrid  man-in-the-loop  simulators, 
such  as  the  Real-Time  Electromagnetic  Digi¬ 
tally  Controlled  Analyzer  and  Processor 
(REDCAP)  and  the  Air  Force  Electronic 
Warfare  Evaluation  Simulator  (AFEWES) 
can  provide  a  capability  to  test  against  new 
threats.  Hybrid  simulators  cost  less  and  are 
safer  than  field  testing.  The  field  test  ranges 
are  used  when  a  wider  range  of  actions  and 
reactions  by  aircraft  and  ground  threat  sys¬ 
tem  operations  is  required. 

Where  one  tool  is  weak,  another  may  be 
strong.  By  using  all  the  tools,  an  EC  tester 
can  do  a  complete  job  of  testing.  An  ex¬ 
ample  of  an  integrated  methodology  is 
shown  in  Figure  20-1.  The  EC  integrated 
testing  can  be  summarized  as: 

(1)  Initial  modeling  phase  for  sensitivity 
analysis  and  test  planning, 

(2)  Active  test  phases  at  hybrid  labora¬ 
tory  simulator  and  field  range  facilities, 

(3)  Test  data  reduction  and  analysis, 

(4)  Post-test  modeling  phase  repeating 
the  first  step  using  test  data  for  extrapola¬ 
tion, 

(5)  Force  effectiveness  modeling  and 
analysis  phase  to  determine  the  incremen¬ 
tal  contribution  of  the  new  system  to  total 
force  effectiveness. 

Another  alternative  is  the  electronic  com¬ 
bat  test  process  proposed  in  the  Air  Force 


Electronic  Combat  Development  Test  Process 
Guide,  May  1991,  issued  by  what  is  now  the 
Air  Staff  T&E  Element,  AF/TE.  The  six  step 
process  described  here  is  graphically  rep¬ 
resented  by  Figure  20-2: 

(1)  Deriving  test  requirements, 

(2)  Conducting  pretest  analysis  to  pre¬ 
dict  EC  system  performance, 

(3)  Conducting  test  sequences  under  pro¬ 
gressively  more  rigorous  ground-  and 
flight-test  conditions, 

(4)  Processing  test  data, 

(5)  Conducting  post-test  analysis  and 
evaluation  of  operational  effectiveness  and 
suitability, 

(6)  Feeding  results  back  to  the  system; 
development  employment  process. 

As  can  be  seen  from  Figure  20-3,  assuming 
a  limited  budget  and  field  test  being  the 
most  expensive  per  number  of  trials,  the 
cost  of  test  trials  forces  the  developer  and 
tester  to  make  tradeoffs  to  obtain  the  neces¬ 
sary  test  data.  Many  more  iterations  of  a 
computer  simulation  can  be  run  for  the  cost 
of  an  open-air  test. 

20.3  TESTING  OF  C^ISR 
SYSTEMS 

20.3.1  Special  Considerations 
When  Testing  C^ISR  Systems 

The  purpose  of  a  C^ISR  system  is  to  provide 
a  commander  with  timely  and  relevant  in¬ 
formation  to  support  sound  war  fighting 
decision-making.  A  variety  of  problems 
face  the  C^ISR  system  tester.  However,  in 
evaluating  command  effectiveness,  it  is 
difficult  to  separate  the  contribution  made 
by  the  CflSR  system  from  the  contribution 
made  by  the  commander's  innate,  cogni- 
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Figure  20-1 .  Integrated  EC  Testing  Approach 
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Figure  20-3.  EC  Test  Resource  Categories 


tive  processes.  To  assess  a  OISR  system  in 
its  operational  environment,  it  must  be  con¬ 
nected  to  the  other  systems  with  which  it 
would  normally  operate,  making  traceabil¬ 
ity  of  test  results  difficult.  Additionally, 
modern  C^ISR  systems  are  software  inten¬ 
sive  and  highly  interactive,  with  complex 
man-machine  interfaces.  Measuring  OISR 
system  effectiveness  thus  requires  the  tester 
to  use  properly  trained  user  troops  during 
the  test  and  to  closely  monitor  software  test 
and  evaluation  (T&E).  The  OISR  systems 
of  defense  agencies  and  the  Services  (Army, 
Navy,  Air  Force  and  Marines)  are  expected 
to  interoperate  with  each  other  and  with 
those  of  the  NATO  forces;  hence,  the  tester 
must  also  ensure  inter-Service  and  NATO 
compatibility,  interoperability  and  integra¬ 
tion. 

20.3.2  OI  Test  Facilities 

Testing  of  Command,  Control,  Commu¬ 
nications,  Computer  and  Intelligence  (C^I) 
systems  will  have  to  rely  more  on  the  use 
of  computer  simulations  and  C^I  test-beds 
to  assess  their  overall  effectiveness.  The 
Defense  Information  Systems  Agency  is 
responsible  for  ensuring  interoperability 
among  all  U.S.  tactical  C^I  systems  that 
would  be  used  in  joint  or  combined  op¬ 
erations,  directs  the  Joint  Interoperability 
Test  Command  (JITC)  at  Ft.  Huachuca, 
AZ.  The  JITC  is  a  test-bed  for  C^I  systems 
compatibility,  interoperability,  and  inte¬ 
gration. 

20.4  TRENDS  IN  TESTING 
ai  SYSTEMS 

21.4.1  Evolutionary  Acquisition 
of  C^I  Systems 

Evolutionary  Acquisition  (E A)  is  a  strategy 
designed  to  provide  an  early,  useful  capa¬ 
bility  even  though  detailed  overall  system 
requirements  cannot  be  fully  defined  at 


the  program's  inception.  The  EA  strategy 
contributes  to  a  reduction  in  the  risks  in¬ 
volved  in  system  acquisition,  since  the  sys¬ 
tem  is  developed  and  tested  in  manageable 
increments.  The  C^I  systems  are  likely  can¬ 
didates  for  EA  because  they  are  character¬ 
ized  by  system  requirements  that  are  diffi¬ 
cult  to  quantify  or  even  articulate  and  that 
are  expected  to  change  as  a  function  of 
scenario,  mission,  theater,  threat  and  emerg¬ 
ing  technology.  Therefore,  the  risk  associ¬ 
ated  with  developing  these  systems  can  be 
very  great. 

Studies  by  the  Defense  Systems  Manage¬ 
ment  College  (Reference  38)  and  the  Inter¬ 
national  Test  and  Evaluation  Association 
(ITEA)  have  addressed  the  issues  involved 
in  the  EA  and  testing  of  Command,  Con¬ 
trol,  and  Communications  (C^)  systems. 
The  ITEA  study  illustrated  E A  in  Figure  20- 
4  and  stated  that: 

With  regard  to  the  tester's  role  in  EA, 
the  study  group  concluded  that  itera¬ 
tive  test  and  evaluation  is  essential  for 
success  in  an  evolutionary  acquisition. 
The  tester  must  become  involved 
early  in  the  acquisition  process  and 
contribute  throughout  the  develop¬ 
ment  and  fielding  of  the  core  and  the 
subsequent  increments.... The  testers 
contribute  to  the  requirements  pro¬ 
cess  through  feedback  of  test  results 
to  the  user .. .and . . .must  judge  the  abil¬ 
ity  of  the  system  to  evolve  (Reference 
115). 

The  testing  of  EA  systems  presents  the 
tester  with  a  unique  challenge  as  the  core 
system  must  be  tested  during  fielding  and 
the  first  increment  before  the  core  testing  is 
completed.  This  could  lead  to  a  situation 
where  the  tester  has  three  or  four  tests 
ongoing  on  various  increments  of  the  same 
system.  The  program  manager  (PM)  must 
insist  that  the  testing  for  EA  systems  be 
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carefully  planned  to  ensure  the  test  data  is 
shared  by  all  and  there  is  a  minimum  of 
repetition  or  duplication  in  testing. 

20.4.2  Radio  Vulnerability 

The  Radio  Vulnerability  Analysis  (RVAN) 
methodology  is  for  assessing  the  anti-jam 
capability  and  limitations  of  radio  fre¬ 
quency  (RF)  data  links  when  operating  in  a 
hostile  electronic  countermeasures  (ECM) 
environment.  The  RVAN  evolved  from  the 
test  methodologies  developed  for  an  Office 
of  the  Secretary  of  Defense  (OSD)-char- 
tered  Joint  Test  on  Data  Link  Vulnerability 
Analysis  (DVAL).  In  1983,  OSD  directed 
the  Services  to  apply  the  DVAL  methodol¬ 
ogy  to  all  new  data  links  being  developed. 

The  purpose  of  the  DVAL  methodology  is 
to  identify  and  quantify  the  anti-jam  capa¬ 
bilities  and  vulnerabilities  of  a  RF  data  link 
operating  in  a  hostile  ECM  environment. 
The  methodology  is  applied  throughout 
the  acquisition  process  and  permits  early 
identification  of  needed  design  modifica¬ 
tions  to  reduce  identified  ECM  vulnerabili¬ 
ties.  The  following  four  components  deter¬ 
mine  a  data  link's  electronic  warfare  (EW) 
vulnerability: 

(1)  The  susceptibility  of  a  data  link;  i.e., 
the  receiver's  performance  when  subjected 
to  intentional  threat  ECM; 

(2)  The  interceptibility  of  the  data  link; 
i.e.,  the  degree  to  which  the  transmitter 
could  be  intercepted  by  enemy  intercept 
equipment; 

(3)  The  accessibility  of  the  data  link;  i.e., 
the  likelihood  that  a  threat  jammer  could 
degrade  the  data  link's  performance; 

(4)  The  feasibility  that  the  enemy  would 
intercept  and  jam  the  data  link  and  success¬ 
fully  degrade  its  performance. 


The  analyst  applying  the  DVAL  methodol¬ 
ogy  will  require  test  data;  and  the  test  man¬ 
ager  of  the  Command,  Control,  Communi¬ 
cations,  and  Intelligence  (C^I)  system,  of 
which  the  data  link  is  a  component,  will  be 
required  to  provide  this  data.  The  DVAL 
joint  test  methodologies  and  test  results  are 
onfile  as  part  of  theJointTest  Library  being 
maintained  by  the  USAF  Operational  Test 
and  Evaluation  Center,  Kirtland  AFB,  NM. 

20.5  T&E  of  Surveillance  and 
Reconnaissance  systems 

Intelligence,  Surveillance,  and  Reconnais¬ 
sance  (ISR)  capabilities  provide  the  requi¬ 
site  battlespace  awareness  tools  for  U.S. 
Forces  to  take  and  hold  the  initiative,  in¬ 
crease  operating  tempo,  and  concentrate 
power  at  the  time  and  place  of  their  choos¬ 
ing.  These  vital  capabilities  are  achieved 
through  highly  classified  sensor  systems 
ranging  from  satellites,  aircraft,  maritime 
vessels,  electronic  intercept,  and  the  sol¬ 
dier  in  the  field  to  the  systems  required  to 
analyze  that  data,  synthesize  it  into  useable 
information,  and  disseminate  that  infor¬ 
mation  ina  timely  fashion  to  the  warfighter. 
As  a  general  rule,  ISR  systems  are  consid¬ 
ered  to  be  Joint  Systems. 

Because  of  the  multifaceted  nature  of  ISR 
programs,  the  classified  nature  of  how  data 
is  gathered  in  the  operational  element,  test 
planning  to  verify  effectiveness  and  suit¬ 
ability  can  be  complex.  Adding  to  that  in¬ 
herent  complexity  is  the  variable  nature  of 
organizational  guidance  directive  upon  the 
tester.  While  the  broad  management  prin¬ 
ciples  enunciated  by  DoD  5000.1  will  apply 
to  highly  sensitive  classified  systems  and 
cryptographic  and  intelligence  programs, 
the  detailed  guidance  contained  in  DoD 
5000.2R  strictly  applies  only  to  MDAPs  and 
MAISs.  Many  ISR  programs  fall  below  this 
threshold  and  the  wise  test  manager  should 
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anticipate  that  several  agencies  will  have 
taken  advantage  of  this  opening  to  tailor 
organizational  guidance. 

Key  issues  for  the  test  and  evaluation  of  ISR 
systems  to  consider  include  compliance 
verification  with  the  Compatibility,  Inte¬ 
gration,  and  Interoperability  (CII)  require¬ 
ments  contained  in  DoDD  4630.5,  CJCSI 
3170.01,  and  CJCSI  6212.01  A  as  certified  by 
the  Joint  Interoperability  Test  Command 
(JITC);  completion  of  the  system  security 
certification  required  prior  to  processing 
classified  or  sensitive  material,  verification 
and  documentation  of  required  support 
from  interfaced  C^ISR  systems  in  the  C^I 
Support  Plan  to  ensure  the  availability  and 
quality  of  required  input  data,  character¬ 
ization  of  the  maturity  of  mission  critical 
software,  finalization  of  the  range  of  hu¬ 
man  factors  analysis,  and  consideration  of 
Information  Operations  vulnerabilities/ 
capabilities.  In  addition  to  this  partial  list¬ 
ing,  many  of  these  systems  will  operate 
inside  a  matrix  of  ISR  system  architectures 
which  must  be  carefully  considered  for  test 


planning  purposes.  As  a  final  issue.  Ad¬ 
vanced  Concept  Technology  Demonstra¬ 
tion  programs  are  being  used  to  quickly 
deliver  capability  to  the  user  because  of  the 
critical  and  time  sensitive  nature  of  many 
ISR  requirements.  The  test  manager  must 
carefully  consider  structuring  the  T&E  ef¬ 
fort  in  light  of  the  inherent  nature  of  ACDT 
programs. 

20.6  SUMMARY 

The  EC  systems  must  be  tested  under  con¬ 
ditions  representative  of  the  dense  threat 
signal  environments  in  which  they  will 
operate.  The  C^ISR  systems  must  be  tested 
in  representative  environments  where  their 
interaction  and  responsiveness  can  be  dem¬ 
onstrated.  The  solution  for  the  tester  is  an 
integrated  approach  using  a  combination 
of  analytical  models,  computer  simulations, 
hybrid  laboratory  simulators  and  testbeds, 
and  actual  field  testing.  The  tester  must 
understand  these  test  techniques  and  re¬ 
sources  and  apply  them  in  EC  and  C'*ISR 
test  and  evaluation. 
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21.1  INTRODUCTION 

This  chapter  discusses  the  planning  and 
management  of  a  multi-Service  test  pro¬ 
gram.  A  multi-Service  test  program  is  con¬ 
ducted  when  a  system  is  to  be  acquired  for 
use  by  more  than  one  Service  or  when  a 
system  must  interface  with  equipment  of 
another  Service.  A  multi-Service  test  pro¬ 
gram  should  not  be  confused  with  the  Of¬ 
fice  of  the  Secretary  of  Defense  (OSD)-spon- 
sored,  nonacquisition-oriented  Joint  Test 
and  Evaluation  (JT&E)  program  (Depart¬ 
ment  of  Defense  (DoD)  5000.3-M-4).  A  brief 
description  of  the  JT&E  program  is  pro¬ 
vided  in  Chapter  6. 

21.2  BACKGROUND 

Formulation  of  a  multi-Service  test  and 
evaluation  (T&E)  program  designates  the 
participants  in  the  program  and  gives  a 
Lead  Service  responsibility  for  preparing  a 
single  report  concerning  a  system's  opera¬ 
tional  effectiveness  and  suitability.  (The 
Lead  Service  is  the  Service  responsible  for 
the  overall  management  of  a  Joint  Acquisi¬ 
tion  program.  A  "Supporting  Service"  is  a 
Service  designated  as  a  participant  in  the 
system  acquisition.) 

A  multi-Service  T&E  program  may  include 
either  development  test  and  evaluation 
(DT&E)  or  operational  test  and  evaluation 
(OT&E)  or  both.  The  Service's  operational 
test  agencies  have  executed  a  formal  Memo¬ 
randum  of  Agreement  on  multi-Service 
OT&E  (Reference  35)  that  provides  a  frame¬ 
work  for  the  conduct  of  a  multi-Service 
operational  test  program. 


Air  Force  Instruction  99-100  series  and  the 
Army  DA  Pam  73-xx  series  provide  guid¬ 
ance  for  procedures  followed  in  a  multi- 
Service  T&E  program.  Generally  the  pro¬ 
cess  includes: 

(1)  In  a  multi-Service  acquisition  pro¬ 
gram,  T&E  is  planned  and  conducted 
according  to  Lead  Service  regulations. 
The  designated  Lead  Service  will  have 
the  overall  responsibility  for  manage¬ 
ment  of  the  multi-Service  program  and 
will  ensure  that  supporting  service 
requirements  are  included.  If  another 
Service  has  certain  unique  T&E  re¬ 
quirements,  testing  for  these  unique 
requirements  may  be  planned,  funded, 
and  conducted  according  to  that 
Service's  regulations. 

(2)  Participating  Services  will  pre¬ 
pare  reports  in  accordance  with  their 
respective  regulations.  The  Lead  Ser¬ 
vice  will  prepare  and  coordinate  a 
single  DT&E  report  and  a  single 
OT&E  report,  which  will  summarize 
the  conclusions  and  recommenda¬ 
tions  of  each  Service's  reports.  Ratio¬ 
nale  will  be  provided  to  explain  any 
significant  differences.  The  indi¬ 
vidual  Service  reports  may  be  at¬ 
tached  to  this  single  report. 

(3)  Deviations  from  the  Lead  Service 
T&E  regulations  may  be  accommo¬ 
dated  by  mutual  agreement  among 
the  Services  involved. 


21.3  TEST  PROGRAM 
RESPONSIBILITIES 

The  Lead  Service  has  overall  management 
responsibility  for  the  program.  It  must  en¬ 
sure  that  supporting  ^rvice  requirements 
are  included  in  the  formulation  of  the  basic 
resource  and  planning  documents. 

A  multi-Service  Test  and  Evaluation  In¬ 
tegrated  Product  Team  (TE  IPT)  should 
be  established  for  each  multi-Service  test 
program.  Its  membership  consists  of  one 
senior  representative  from  each  partici¬ 
pating  Service  or  agency  headquarters. 
The  TE  IPT  works  closely  with  the  pro¬ 
gram  management  office  (PMO)  and  is 
responsible  for  arbitrating  all  disagree¬ 
ments  among  Services  that  cannot  be  re¬ 
solved  at  the  working  level. 

Resource  requirements  are  documented  in 
the  Test  and  Evaluation  Master  Plan 
(TEMP).  Each  participating  Service  is  di¬ 
rected  to  budget  for  the  testing  necessary  to 
accomplish  its  assigned  test  objectives  and 
for  the  participation  of  its  personnel  and 
equipment  in  the  entire  test  program.  Sepa¬ 
rate  annexes  may  be  used  to  address  each 
Service's  test  requirements. 

21.4  TEST  TEAM  STRUCTURE 

A  sample  test  team  structure  is  shown  in 
Figure  21-1 .  As  shown  in  the  figure.  Service 
test  teams  work  through  a  Service  deputy 
test  director  or  senior  representative.  The 
test  director  exercises  test  management 
authority  but  not  operational  control  over 
the  test  teams.  The  responsibilities  include 
integration  of  test  requirements  and  effi¬ 
cient  scheduling  of  test  events.  The  deputy 
test  directors  exercise  operational  control 
or  test  management  authority  over  their 
Service  test  teams  in  accordance  with  their 
Service  directives.  Additionally,  they  act  as 
advisers  to  the  test  director;  represent  their 


Service's  interests;  and  are  responsible,  at 
least  administratively,  for  resources  and 
personnel  provided  by  their  Services. 

21.5  TEST  PLANNING 

Test  planning  for  multi-Service  T&E  is  ac¬ 
complished  in  the  manner  prescribed  by 
Lead  Service  directions  and  in  accordance 
with  the  following  general  procedures  ex¬ 
tracted  from  the  "Memorandum  of  Agree¬ 
ment  on  Multi-Service  OT&E  and  Joint 
T&E": 

(1)  The  Lead  Service  T&E  agency  begins 
the  planning  process  by  issuing  a  call  to  the 
supporting  Service  T&E  agencies  for  criti¬ 
cal  issues  and  test  objectives. 

(2)  The  Lead  Service  T&E  agency  con¬ 
solidates  the  objectives  into  a  list  and  coor¬ 
dinates  the  list  with  the  supporting  Service 
T&E  agencies. 

(3)  The  Lead  Service  T&E  agency  accom¬ 
modates  supporting  Service  T&E  require¬ 
ments  and  input  in  the  formal  coordination 
action  of  the  TEMP. 

(4)  Participating  T&E  agency  project  of¬ 
ficers  assign  responsibility  for  the  accom¬ 
plishment  of  test  objectives  (from  the  con¬ 
solidated  list)  to  each  T&E  agency.  These 
assignments  are  made  in  a  mutually  agree¬ 
able  manner.  Each  agency  is  then  respon¬ 
sible  for  resource  identification  and  ac¬ 
complishment  of  its  assigned  test  objec¬ 
tives  under  the  direction  of  the  Lead  Ser¬ 
vice  T&E  agency. 

(5)  Each  participating  agency  prepares 
the  portion  of  the  overall  test  plan(s)  for 
its  assigned  objectives,  in  the  Lead  Ser¬ 
vice  test  plan(s)  format,  and  identifies  its 
data  needs. 

(6)  The  Lead  Service  T&E  agency  pre¬ 
pares  the  multi-Service  T&E  test  plan(s). 
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Figure  21-1 .  Simple  Multi-Service  OT&E  Test  Team  Composition 


consolidating  the  input  from  all  participat¬ 
ing  agencies. 

21.6  DISCREPANCY  REPORTING 

In  a  multi-Service  T&E  program,  a  discrep¬ 
ancy  report  is  a  report  of  any  condition  that 
reflects  adversely  on  the  item  being  tested 
and  that  must  be  reported  outside  the  test 
team  for  corrective  action.  The  discrepancy 
reporting  system  of  the  Lead  Service  is 
normally  used.  All  members  of  the  multi- 
Service  test  team  will  report  discrepancies 
through  their  Service's  system. 

Items  undergoing  test  will  not  necessarily 
be  used  by  each  of  the  Services  for  identical 
purposes.  As  a  result,  a  discrepancy  con¬ 
sidered  disqualifying  by  one  Service  is  not 
necessarily  disqualifying  for  all  Services. 
Discrepancy  reports  of  a  disqualifying  na¬ 
ture  must  include  a  statement  by  the  con¬ 
cerned  Service  of  why  the  discrepancy  has 
been  so  classified.  It  also  includes  state¬ 
ments  by  the  other  Services  as  to  whether 
or  not  the  discrepancy  affects  them  signifi¬ 
cantly. 

If  one  of  the  participating  Services  identi¬ 
fies  a  discrepancy  that  it  considers  as  war¬ 
ranting  termination  of  the  test,  the  circum¬ 
stances  are  reported  immediately  to  the 
test  director. 

22.7  TEST  REPORTING 

The  following  test-reporting  policy  ap¬ 
plies  to  multi-Service  OT&E  programs: 

(1)  Interim  test  reports  may  be  prepared 
to  support  program  reviews.  If  they  are 
required  on  a  particular  program;  they  are 
prepared  in  accordance  with  Lead  Service 
directives  and  coordinated  with  all  partici¬ 
pating  OT&E  agencies  prior  to  release. 


(2)  Within  60  days  of  the  end  of  testing, 
the  multi-Service  OT&E  team  must  present 
a  factual  report  of  the  test  to  all  participat¬ 
ing  OT&E  agencies.  (This  factual  report 
presents  the  data  collected  but  no  evalua¬ 
tion,  conclusions  or  recommendations  con¬ 
cerning  the  data.) 

(3)  Each  participating  OT&E  agency  pre¬ 
pares  an  independent  evaluation  report  in 
its  own  format  and  forwards  that  report 
through  its  normal  Service  charmels. 

(4)  Approved  independent  evaluation 
reports  are  distributed  to  all  participating 
OT&E  agencies. 

(5)  The  Lead  Service  OT&E  agency  is 
responsible  for  preparing  the  Defense  Ac¬ 
quisition  Board  (DAB)  briefing(s)  which  is 
(are)  coordinated  with  all  participating 
OT&E  agencies. 

22.8  SUMMARY 

Multi-Service  test  programs  are  conducted 
by  two  or  more  Services  when  a  system  is  to 
be  acquired  for  employment  by  more  than 
one  Service  or  when  a  system  must  inter¬ 
face  with  equipment  of  another  Service. 
Test  procedures  for  multi-Service  T&E  fol¬ 
low  those  of  the  designated  Lead  Service, 
with  mutual  agreements  resolving  areas 
where  deviations  are  necessary.  Care  must 
be  exercised  when  integrating  test  results 
and  reporting  discrepancies  since  items 
undergoing  testing  may  be  used  for  differ¬ 
ent  purposes  in  different  Services.  Close 
coordination  is  required  to  ensure  that  an 
accurate  summary  of  the  developing 
system's  capabilities  is  provided  to  Service 
and  DoD  decision  authorities. 
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INTERNATIONAL  TEST 
AND  EVALUATION  PROGRAMS 


22.1  INTRODUCTION 

This  chapter  discusses  test  and  evaluation 
(T&E)  from  an  international  perspective.  It 
describes  the  Office  of  the  Secretary  of  De¬ 
fense  (OSD)-sponsored  Foreign  Compara¬ 
tive  Test  (FCT)  Program  (10  U.S.C.  2350) 
and  the  International  Test  Operations  Pro¬ 
cedures.  Factors  that  bear  on  the  T&E  of 
multinational  acquisition  programs  are  dis¬ 
cussed  also. 

22.2  FOREIGN  COMPARATIVE 
TEST  PROGRAM 

22.2.1  Program  Objective 

The  FCT  Program  is  designed  to  support 
the  evaluation  of  a  foreign  nation's  weap¬ 
ons  system,  equipment  or  technology  in 
terms  of  its  potential  to  meet  a  valid  re¬ 
quirement  of  one  or  more  of  the  U.S. 
Armed  Services.  Additional  goals  of  the 
FCT  program  include  avoiding  unneces¬ 
sary  duplication  in  development,  enhanc¬ 
ing  standardization  and  interoperability 
and  promoting  international  technology 
exchanges.  The  FCT  program  is  not  in¬ 
tended  for  use  in  exploiting  threat  systems 
or  for  intelligence  gathering.  The  primary 
objective  of  the  program  is  to  support  the 
U.S.  national  policy  of  encouraging  inter¬ 
national  armaments  cooperation  and  to 
reduce  the  costs  of  research  and  develop¬ 
ment.  Policy  and  procedures  for  the  execu¬ 
tion  of  the  program  are  documented  in 
Department  of  Defense  (DoD)  5000.3-M-2. 


22.2.2  Program  Administration 

Foreign  weapons  evaluation  activities  and 
responsibilities  are  assigned  to  the  Direc¬ 
tor,  Test,  Systems  Engineering,  and  Evalu¬ 
ation  (DTSEE)  by  direction  of  the  Congress 
in  1980.  Each  year,  sponsoring  military  ser¬ 
vices  forward  Candidate  Nomination  Pro¬ 
posals  (CNPs)  for  systems  to  be  evaluated 
under  the  FCT  program  to  the  DTSEE.  The 
Services  are  encouraged  to  prepare  and 
submit  a  CNP  whenever  a  promising  can¬ 
didate  that  appears  to  satisfy  a  current  or 
potential  Service  requirement  is  found.  A 
CNP  must  contain  the  information  as  re¬ 
quired  by  DoD  5000.3-M-2. 

The  fundamental  criterion  for  FCT  pro¬ 
gram  selection  is  the  candidate  system's 
potential  to  satisfy  operational  or  training 
requirements  that  exist  or  are  projected.  Its 
possible  contribution  to  the  U.S.  technol¬ 
ogy  base  is  considered  also.  Additional 
factors  influencing  candidate  selection  in¬ 
clude:  candidate  maturity,  available  test 
data,  multi-Service  interest,  existence  of  a 
statement  of  operational  requirement  need, 
potential  for  subsequent  procurement, 
sponsorship  by  U.S.-based  licensee,  realis¬ 
tic  evaluation  schedule  cost,  DoD  compo¬ 
nent  OSD  evaluation  cost-sharing  proposal, 
and  preprogrammed  procurement  funds. 
For  technology  evaluation  programs  within 
the  FCT  program,  the  candidate  nomina¬ 
tion  proposal  must  address  the  specific 
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arrangements  under  which  the  United 
States  and  foreign  participants  (govern¬ 
ments,  armed  forces,  corporations)  will 
operate.  These  may  include  govemment- 
to-government  Memoranda  of  Agreement, 
private  industry  licensing  agreements,  data 
exchange  agreements  and/or  cooperative 
technology  exchange  programs. 

Foreign  weapons  evaluation  activities  are 
funded  by  d^D  and  executed  by  the  Ser¬ 
vice  with  the  potential  need  for  the  system. 
Points  of  contact  at  the  headquarters  level 
in  each  Service  monitor  the  conduct  of  the 
programs.  Work  is  performed  in  laborato¬ 
ries  and  test  centers  throughout  the  coun¬ 
try.  Systems  evaluated  recently  under  the 
FCT program  include  millimeter  wave  com¬ 
munications  equipment,  chemical  defense 
equipment,  gunnery  devices,  maritime  de¬ 
coys  and  navigational  systems.  The  DTSEE 
shall  notify  Congress  a  minimum  of  thirty 
days  prior  to  the  commitment  of  funds  for 
initiation  of  new  FCT  evaluations. 

22.3  NATO  COMPARATIVE 
TEST  PROGRAM 

The  NATO  Comparative  Test  Program  has 
been  integrated  with  the  FCT  program.  It 
was  created  by  an  act  of  the  Congress  in  the 
fiscal  year  (FY)86  Defense  Authorization 
Bill.  The  program  supported  the  evalua¬ 
tion  of  NATO  nations'  weapons  systems, 
equipment  and  technology  and  assessed 
their  suitability  for  use  by  U.S.  forces.  The 
selection  criteria  for  the  NATO  Compara¬ 
tive  Test  Program  were  essentially  the  same 
as  for  the  FCT  program.  The  exception  was 
that  the  equipment  must  be  produced  by  a 
NATO  member  nation  and  be  considered 
as  an  alternative  to  a  system  that  was  either 
in  a  late  stage  of  development  in  the  United 
States  or  that  offered  a  cost,  schedule  or 
performance  advantage  over  U.S.  equip¬ 
ment.  In  addition,  the  NATO  Comparative 
Test  Program  required  that  notification  be 


sent  to  the  Armed  Services  and  Appropria¬ 
tions  Committees  of  the  House  of  Repre¬ 
sentatives  and  Senate  before  funds  were 
obligated.  With  this  exception,  the  NATO 
Comparative  Test  Program  followed  the 
same  nomination  process  and  administra¬ 
tive  procedures.  Guidelines  for  the  pro¬ 
gram  were  also  contained  in  DoD  5000.3- 
M-2. 

Examples  of  proposals  funded  under  the 
NATCDComparative  Test  Program  included 
T&E  of  a  German  mine  reconnaissance  and 
detection  system  for  the  Army,  a  United 
Kingdom-designed  minehunter  for  the 
Navy,  and  the  Norwegian  Penguin  missile 
system  for  the  Air  Force.  According  to  the 
1^88  Report  of  the  Secretary  of  Defense  to 
the  Congress,  the  program  generated  con¬ 
siderable  interest  among  NATO  allied  na¬ 
tions  and  became  a  primary  way  of  pro¬ 
moting  armaments  cooperation  within 
NATO. 

Problems  associated  with  testing  foreign 
weapons  normally  stem  from  politics,  na¬ 
tional  pride  and  a  lack  of  previous  test  data . 
When  foreign  companies  introduce  weapon 
systems  for  testing,  they  often  will  attempt 
to  align  the  U.S.  military /congressional  or¬ 
ganizations  with  their  systems.  For  ex¬ 
ample,  when  a  foreign  nation  introduced 
an  antitank  weapon  to  the  Army,  they  did 
so  by  having  a  U.S.  Senator  write  the  Army 
stating  a  need  for  the  system.  Attached  to 
the  letter  was  a  document  containing  doc¬ 
trine  to  employ  the  system  and  a  test  con¬ 
cept  to  use  when  evaluating  the  system. 
Systems  tested  in  the  NATO  Comparative 
Test  Program  often  become  involved  in 
national  pride.  The  test  community  must 
be  careful  not  to  allow  national  pride  to  be 
a  driving  force  in  the  evaluation.  At  times, 
the  9mm  pistol  competition  in  NATO  re¬ 
sembled  an  international  soccer  match,  with 
each  competing  nation  cheering  for  their 
pistol  and  many  other  nations  selecting 
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sides.  Evaluating  the  9mm  pistol  was  diffi¬ 
cult  because  of  these  forces.  Thus,  U.S. 
testers  must  make  every  effort  to  obtain  all 
available  test  data  on  foreign  systems.  These 
data  can  be  used  to  help  validate  the  evolv¬ 
ing  test  data  and  additional  test  data  dur¬ 
ing  the  evaluation. 

22.4  T&E  MANAGEMENT  IN 
MULTINATIONAL  PROGRAMS 

22.4.1  Compatibility  With  Allies 

Rationalization,  standardization  and 
interoperability  have  become  increasingly 
important  elements  in  the  materiel  acquisi¬ 
tion  process.  Public  Law  94-361,  passed  on 
July  14, 1976,  requires  that  "equipment  for 
use  of  personnel  of  the  Armed  Forces  of  the 
United  States  stationed  in  Europe  under 
the  terms  of  the  North  Atlantic  Treaty 
should  be  standardized  or  at  least 
interoperable  with  equipment  of  other 
members  of  the  North  Atlantic  Treaty  Or¬ 
ganization".  Program  Managers  (PM)  and 
test  managers  must,  therefore,  be  fully 
aware  of  any  potential  international  appli¬ 
cations  of  the  systems  for  which  they  are 
responsible.  The  Joint  Logistics  Commanders 
Guide  for  the  Management  of  Multinational 
Programs  published  by  the  Defense  Sys¬ 
tems  Management  College  (Reference  47) 
is  a  valuable  compendium  of  information 
for  the  PM  of  a  developing  system  with 
potential  multinational  applications. 

Representatives  of  the  United  States,  United 
Kingdom,  France  and  Germany  have  signed 
a  Memorandum  of  Agreement  concerning 
the  mutual  acceptability  of  each  country's 
T&E  data.  This  agreement  seeks  to  avoid 
redundant  testing  by  documenting  the  ex¬ 
tent  of  understanding  among  involved  gov¬ 
ernments  concerning  mutual  acceptability 
of  respective  T&E  procedures  for  systems 
that  are  developed  in  one  country  and  are 
candidates  for  procurement  by  one  or  more 


of  the  other  countries.  Focal  points  for 
development  and  operational  testing  in 
each  of  the  countries  are  identified,  and 
procedures  governing  generation  and  re¬ 
lease  of  T&E  data  are  described  in  the 
Memorandum  of  Understanding  (MOU). 

Early  and  thorough  planning  is  an  impor¬ 
tant  element  of  any  successful  T&E  pro¬ 
gram  but  is  even  more  critical  in  a  multina¬ 
tional  program.  Agreement  must  be  reached 
concerning  T&E  procedures,  data  require¬ 
ments  and  methodology.  Differences  in 
tactics,  battlefield  representations  and  mili¬ 
tary  organizations  may  make  it  difficult  for 
one  nation  to  accept  another's  test  data. 
Therefore,  agreement  must  be  reached  in 
advance  concerning  the  operational  test 
scenario  and  battlefield  representation  that 
will  be  used. 

22.4.2  International  Test  Operations 
Procedures 

The  International  Test  Operations  Proce¬ 
dures  (ITOPs)  are  documents  containing 
standardized  state-of-the-art  test  proce¬ 
dures  prepared  by  the  cooperative  efforts 
of  France,  Germany,  the  U.K.  and  the  U.S. 
Their  use  assures  high  quality,  efficient, 
accurate,  and  cost  effective  testing.  The 
DTSEE  is  the  OSD  sponsor  for  providing 
basic  guidance  and  direction  to  the  ITOPs 
processes.  The  ITOPs  program  is  intended 
to  shorten  and  reduce  costs  of  the  materiel 
development  and  acquisition  cycle,  mini¬ 
mize  duplicate  testing,  improve 
interoperability  of  U.S.  and  Allied  equip¬ 
ment,  promote  the  cooperative  develop¬ 
ment  and  exchange  of  advanced  test  tech¬ 
nology,  and  expand  the  customer  base.  Each 
Service  has  designated  an  ITOPs  point  of 
contact.  The  Army  uses  the  Test  and  Evalu¬ 
ation  Management  Agency  (TEMA),  in  the 
Navy  it  is  the  Director,  Navy  T&E  Division 
(N-91),  and  the  Air  Force  has  the  Chief, 
Policy  and  Program  Division  (AF/TEP). 
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The  Army,  who  initiated  the  program  in 
1979,  is  the  lead  Service.  A  total  of  75  ITOPs 
have  been  completed  and  published  in  six 
technical  areas  under  the  Four-Nation  Test 
and  Evaluation  MOU.  Additional  ITOPs 
are  under  development  by  active  working 
committees.  Completed  documents  are 
submitted  to  the  Defense  Technical  Infor¬ 
mation  Center  (DTIC)  for  official  distribu¬ 
tion. 

22.5  U.S.  AND  NATO  ACQUISITION 
PROGRAMS 

Some  test  programs  involve  combined  de¬ 
velopment  and  test  of  new  weapon  sys¬ 
tems  for  U.S.  and  other  NATO  countries.  In 
these  programs,  some  differences  from  the 
regular  "way  of  doing  things"  occur.  For 
example,  the  formulation  of  the  Request 
for  Proposal  (RFP)  must  be  coordinated 
with  the  North  Atlantic  Program  Manage¬ 
ment  Agency  (NAPMA);  and  their  input  to 
the  Statement  of  Work,  data  requirements, 
operational  test  planning  and  test  schedule 
formulation  must  be  included.  Also,  the 
U.S.  Army  operational  user.  Forces  Com¬ 
mand,  must  be  involved  in  the  operational 
test  program.  Usually,  a  Multinational 
Memorandum  of  Understanding  (MMOU) 
is  created  concerning  test  program  and  pro¬ 
duction  funding,  test  resources,  test  team 
composition,  use  of  national  assets  for  test¬ 
ing,  etc. 

Nations  are  encouraged  to  use  the  data  that 
another  nation  has  gathered  on  similar  test 
programs  to  avoid  duplication  of  effort. 
For  example,  during  the  U.S.  and  NATO 


Airborne  Warning  and  Control  System 
(AW ACS)  ESM  Program,  both  U.S.  and 
NATO  E-3As  will  be  used  for  test  aircraft  in 
combined  development  test  and  evalua¬ 
tion  (DT&E)  and  subsequent  operational 
test  and  evaluation  (OT&E).  Testing  will  be 
conducted  in  the  U.S.  and  European  the¬ 
aters.  The  Joint  Test  Force  will  be  com¬ 
posed  of  program  management  office,  con¬ 
tractor,  U.S.  operational  users.  Air  Force 
Operational  Test  and  Evaluation  Center 
(AFOTEC),  Force  Command  (NATO  us¬ 
ers),  and  logistics  personnel  for  this  pro¬ 
gram.  A  Multinational  Memorandum  of 
Agreement  for  this  program  was  created. 
The  U.S.  program  is  managed  by  the  AW  ACS 
System  Program  Office,  and  the  NATO  pro¬ 
gram  is  managed  by  the  NAPMA. 

22.6  SUMMARY 

The  procurement  of  weapon  systems  from 
foreign  nations  for  use  by  U.S.  Armed  Forces 
can  provide  the  following  advantages:  re¬ 
duced  research  and  development  costs, 
faster  initial  operational  capability,  im¬ 
proved  interoperability  with  friendly  na¬ 
tions,  and  lower  procurement  costs  because 
of  economies  of  scale.  This  is  normally  a 
preferred  solution  to  user  requirements 
before  attempting  to  start  a  new  develop¬ 
ment.  Testing  such  systems  presents  spe¬ 
cific  challenges  to  accommodate  the  needs 
of  all  users.  Such  testing  requires  careful 
advance  planning  and  systematic  execu¬ 
tion.  Expectations  and  understandings 
must  be  well  documented  at  an  early  stage 
to  ensure  that  the  test  results  have  utility 
for  all  concerned. 
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COMMERCIAL 

AND  NONDEVELOPMENT  ITEMS 


23.1  INTRODUCTION 

Many  options  are  available  when  an  ac¬ 
quisition  strategy  for  a  new  system  is 
chosen.  They  range  from  the  last  option 
of  a  traditional  new  research  and  devel¬ 
opment  program  to  modification  of  the 
existing  system.  Between  these  two  ex¬ 
tremes  are  other  acquisition  strategies 
that  call  for  using  commercial  items  or 
nondevelopment  items  (NDIS)  at  differ¬ 
ent  system  levels,  unmodified  or  rugge- 
dized  to  various  extents.  Figure  23-1 
shows  the  broad  spectrum  of  approaches 
that  can  be  taken  in  a  system  acquisition 
and  provides  examples  of  systems  that 
have  been  developed  using  each  ap¬ 
proach. 

23.1.1  Definitions 

A  commercial  item  is  defined  as  any  item, 
other  than  real  property,  that  is  of  a  type 
customarily  used  for  non-governmental 
purposes  and  that:  (1)  has  been  sold,  leased, 
or  licensed  to  the  general  public;  or,  (2)  has 
been  offered  for  sale,  lease,  or  license  to  the 
general  public;  or  any  item  that  evolved 
through  advances  in  technology  or  perfor¬ 
mance  and  that  is  not  yet  available  in  the 
commercial  marketplace,  but  will  be  avail¬ 
able  in  the  commercial  marketplace  in  time 
to  satisfy  the  delivery  requirements  under 
a  government  solicitation.  Also  included 
in  the  definition  are  services  in  support  of 
a  commercial  item,  or  a  type  offered  and 
sold  competitively  in  substantial  quanti¬ 
ties  in  the  commercial  marketplace  based 


on  established  catalog  or  market  prices 
for  specific  tasks  performed  under  stan¬ 
dard  commercial  terms  and  conditions;  this 
does  not  include  services  that  are  sold  based 
on  hourly  rates  without  an  established  cata¬ 
log  or  market  price  for  a  specific  service 
performed. 

An  NDI  is:  (1)  any  previously  developed 
item  of  supply  used  exclusively  for  govern¬ 
mental  purposes  by  a  Federal  Agency,  a 
State,  or  local  government,  or  a  foreign 
government  with  which  the  U.S.  has  a 
mutual  defense  cooperation  agreement;  (2) 
any  item  described  in  (1)  that  requires  only 
minor  modification  or  modifications  of  the 
type  customarily  available  in  the  commer¬ 
cial  marketplace  in  order  to  meet  the  re¬ 
quirements  of  the  procuring  department  or 
agency;  or  (3)  any  item  described  in  (1)  or 
(2)  solely  because  the  item  is  not  yet  in  use. 

All  such  systems  are  required  to  undergo 
technical  and  operational  test  and  evalua¬ 
tion  (T&E)  before  the  procurement  deci¬ 
sion,  unless  the  decision  authority  makes  a 
definitive  decision  that  previous  testing  or 
other  data  (such  as  user/ market  investiga¬ 
tions)  provide  sufficient  evidence  of  ac¬ 
ceptability  (Reference  16). 

23.1.2  Advantages  and  Disadvantages 
of  Commercial  and  NDI  Approaches 

The  use  of  Commercial  and  NDI  offer  the 
following  advantages: 


•  The  time  to  field  a  system  can  be  greatly 
reduced,  providing  a  quicker  response  to 
the  user’s  needs; 

•  Research  and  development  costs  may 
be  reduced; 

•  State-of-the-art  technology  may  be 
available  sooner. 

Commercial  and  NDI  offers  the  following 
disadvantages: 

•  Acquisitions  are  difficult  to  standard¬ 
ize/integrate  with  the  current  fleet  equip¬ 
ment; 

•  Acquisitions  create  logistics  support 
difficulties; 

•  Acquisitions  tend  not  to  have  compa¬ 
rable  competition;  therefore,  there  are  fewer 
second  sources  available; 

•  With  Commercial  and  NDI  acquisi¬ 
tions,  engineering  and  test  data  often  is  not 
available. 

23.1.3  Application  of  Commercial 
and  NDI 

(1)  Commercial  items  or  NDI  may  be 
used  in  the  same  environment  for  which 
the  items  were  designed.  Such  items  nor¬ 
mally  do  not  require  development  testing 
prior  to  the  production  qualification  test 
except  in  those  cases  where  a  contract  may 
be  awarded  to  a  contractor  who  has  not 
previously  produced  an  acceptable  finished 
product  and  the  item  is  assessed  as  high 
risk.  In  that  case,  preproduction  qualifica¬ 
tion  testing  would  be  required  (Reference 
16). 

(2)  Commercial  items  or  NDI  may  be 
used  in  an  environment  other  than  that  for 
which  the  items  were  designed.  Such  items 


may  require  modifications  in  hardware 
and/or  software.  These  items  require  test¬ 
ing  in  an  operational  environment, 
preproduction  qualification  testing  (if  pre¬ 
vious  testing  resulted  in  item  redesign), 
and  production  qualification  testing. 

Integration  of  commercial  items  or  NDI 
into  a  new  development  system  may  re¬ 
quire  some  regression  testing.  These  efforts 
require  more  extensive  research,  develop¬ 
ment  and  testing  to  achieve  effective  opera¬ 
tion  of  the  desired  system  configuration. 
Testing  required  includes:  feasibility  testing 
in  a  military  environment,  preproduction 
qualification  testing,  hardware/software 
integration  testing,  operational  testing  and 
production  qualification  testing. 

Given  the  variety  of  approaches  that  may 
be  employed,  it  is  imperative  that  the  ac¬ 
quisition  strategy  clearly  specifies,  with  the 
agreement  of  the  testing  authority,  the  level 
of  testing  that  will  be  performed  on  com¬ 
mercial  items  and  NDI  systems  and  the 
environment  in  which  those  systems  will 
be  tested. 

23.2  MARKET  INVESTIGATION 
AND  PROCUREMENT 

A  market  investigation  is  the  central  activ¬ 
ity  leading  to  the  Milestone  I  review  deci¬ 
sion  regarding  the  use  of  an  commercial 
item  or  NDI  acquisition  strategy.  The  pur¬ 
pose  of  the  market  investigation  is  to  deter¬ 
mine  the  nature  of  available  products  and 
the  number  of  potential  vendors.  Market 
investigations  may  vary  from  informal  tele¬ 
phone  inquiries  to  comprehensive  indus¬ 
try-wide  reviews.  During  the  market  in¬ 
vestigation,  sufficient  data  must  be  gath¬ 
ered  to  support  a  definitive  decision,  to 
finalize  the  requirements  and  to  develop 
an  acquisition  strategy  that  is  responsive  to 
these  requirements. 

During  the  Market  Investigation  Phase,  a 
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formal  "request  for  information"  process 
may  be  followed  wherein  a  brief  narra¬ 
tive  description  of  the  requirement  is 
published  and  interested  vendors  are  in¬ 
vited  to  respond.  Test  samples  or  test 
items  may  be  leased  or  purchased  at  this 
time  to  support  the  conduct  of  opera¬ 
tional  suitability  tests,  to  evaluate  the 
ability  of  the  equipment  to  satisfy  the 
requirements  and  to  help  build  the  func¬ 
tional  purchase  description  or  system 
specification.  This  type  of  preliminary 
testing  should  not  be  used  to  select  or 
eliminate  any  particular  vendor  or  prod¬ 
uct  unless  it  is  preceded  by  competitive 
contracting  procedures  (Reference  61). 

It  is  imperative  that  technical  and  opera¬ 
tional  evaluators  become  involved  during 
this  early  stage  of  any  commercial  item  or 
NDI  procurement  and  that  they  perform 
an  early  assessment  of  the  initial  issues. 
The  evaluator  must  also  relate  these  issues 
to  test  and  evaluation  criteria  and  provide 
their  independent  evaluation  plans  and 
reports  to  the  decision  authorities  before 
the  Milestone  I  decision  review. 

23.3  COMMERCIAL  ITEM 
AND  NDI  TESTING 

23.3.1  General  Considerations 

Test  and  evaluation  must  be  considered 
throughout  the  acquisition  of  a  system  that 
involves  commercial  items  and  NDI.  The 
program  manager  (PM)  and  his/her  staff 
must  ensure  that  the  testing  community  is 
fully  involved  in  the  acquisition  from  the 
start.  The  amount  and  level  of  testing  re¬ 
quired  depends  on  the  nature  of  the  com¬ 
mercial  item  or  NDI  and  its  anticipated  use; 
it  should  be  planned  to  support  the  design 
and  decision  process.  At  a  minimum,  T&E 
will  be  conducted  to  verify  integration  and 
interoperability  with  other  system  elements . 
All  commercial  item  and  NDI  modifica¬ 


tions  necessary  to  adapt  them  to  the  weapon 
system  environment  wiU  also  be  subject  to 
T&E.  Available  test  results  from  all  com¬ 
mercial  and  government  sources  will  de¬ 
termine  the  actual  extent  of  testing  neces¬ 
sary.  For  example,  a  commercial  item  or 
NDI  usually  encompasses  a  mature  design. 
The  availability  of  this  mature  design  con¬ 
tributes  to  the  rapid  development  of  the 
logistics  support  system  that  will  be  needed . 
In  addition,  there  are  more  "production" 
items  available  for  use  in  a  test  program. 
The  PM  and  his/her  staff  must  remember 
that  these  systems  also  require  activity  in 
areas  associated  with  traditional  develop¬ 
ment  and  acquisition  programs.  For  ex¬ 
ample,  training  and  maintenance  programs 
and  manuals  must  be  developed;  and  suf¬ 
ficient  time  should  be  allowed  for  their 
preparation. 

When  the  solicitation  package  for  a  com¬ 
mercial  item  or  NDI  acquisition  is  as¬ 
sembled,  the  PM  must  ensure  that  it  in¬ 
cludes  the  following  T&E-related  items: 

(1)  Approved  T&E  issues  and  criteria; 

(2)  A  requirement  that  the  offerer  pro¬ 
vide  a  description  of  the  testing  performed 
by  the  contractor  on  the  system,  including 
test  procedures  followed,  data  and  results 
achieved; 

(3)  Production  qualification  test  and  qual¬ 
ity  conformance  requirements; 

(4)  Acceptance  test  plans  for  the  system 
and  its  components. 

23.3.2  Testing  Before  Milestone  I 

Since  an  important  advantage  of  using 
a  commercial  item  or  NDI  acquisition 
strategy  is  reduced  acquisition  time,  it 
is  important  that  testing  not  be  redundant 
and  that  it  is  limited  to  the  minimum  effort 


23-4 


necessary  to  obtain  the  required  data.  Test¬ 
ing  can  be  minimized  by: 

(1)  Obtaining  and  assessing  contractor 
test  results; 

(2)  Obtaining  usage /failure  data  from 
other  customers; 

(3)  Observing  contractor  testing; 

(4)  Obtaining  test  results  from  indepen¬ 
dent  test  organizations  (e.g..  Under-  writer' s 
Laboratory); 

(5)  Verifying  selected  contractor  test  data. 

If  it  is  determined  that  more  information  is 
needed  after  the  initial  data  collection  from 
the  above  sources,  commercial  items  or 
NDI  candidates  may  be  bought  or  leased, 
and  technical  and  operational  tests  may  be 
conducted. 

23.3.3  Testing  After 
Milestone  I 

All  testing  to  be  conducted  after  the  initial 
milestone  decision  to  proceed  with  the  com¬ 
mercial  item  or  NDI  acquisition  should  be 
described  in  the  Acquisition  Strategy  and 
the  Test  and  Evaluation  Master  Plan.  De¬ 
velopment  testing  is  conducted  only  if  spe¬ 
cific  information  that  cannot  be  satisfied  by 
contractor  or  other  test  data  sources  is 
needed.  Operational  testing  is  conducted 
as  needed.  The  independent  operational 
T&E  agency  should  concur  in  any  deci¬ 
sions  to  limit  or  eliminate  operational  test¬ 
ing. 

Test  and  evaluation  continue  even  after  the 
system  has  been  fielded.  This  testing  takes 
the  form  of  a  follow-on  evaluation  to  vali¬ 
date  and  refine:  operating  and  support  cost 
data;  reliability,  availability,  and  maintain¬ 


ability  characteristics;  logistic  support 
plans;  and  training  requirements,  doctrine 
and  tactics. 

23.4  RESOURCES 
AND  FUNDING 

Programming  and  budgeting  for  a  com¬ 
mercial  item  or  NDI  acquisition  present  a 
special  challenge.  Because  of  the  short  du¬ 
ration  of  the  acquisition  process,  the  stan¬ 
dard  lead  times  required  in  the  normal 
Planning,  Programming,  and  Budgeting 
System  cycle  may  be  unduly  restrictive. 
This  situation  can  be  minimized  through 
careful,  advanced  planning  and,  in  the  case 
of  urgent  requirements,  reprogramming/ 
supplemental  funding  techniques. 

Research,  Development,  Test  and  Evalu¬ 
ation  (RDT&E)  funds  are  normally  used 
to  support  the  conduct  of  the  Market  In¬ 
vestigation  Phase  and  the  purchase  or 
lease  of  candidate  systems /components 
required  for  T&E  purposes.  The  RDT&E 
funds  are  also  used  to  support  T&E  ac¬ 
tivities  such  as:  modification  of  the  test 
article;  purchase  of  specifications,  man¬ 
ufacturer's  publications,  repair  parts,  spe¬ 
cial  tools  and  equipment;  transportation 
of  the  test  article  to  and  from  the  test  site; 
and  training,  salaries  and  temporary  duty 
costs  of  T&E  personnel.  Procurement, 
operations  and  maintenance  funds  are 
usually  used  to  support  production  and 
deployment  costs. 

One  chief  reason  for  using  a  commercial 
item  or  NDI  acquisition  strategy  is  reduced 
overall  cost.  Additional  cost  savings  can  be 
achieved  after  a  contract  has  been  awarded 
if  the  PM  ensures  that  incentives  are  pro¬ 
vided  to  contractors  to  submit  value  engi¬ 
neering  change  proposals  to  the  govern¬ 
ment  when  unnecessary  costs  are  identi¬ 
fied. 
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23.5  SUMMARY 

The  use  of  commercial  items  and  NDIs  in  a 
system  acquisition  can  provide  consider¬ 
able  time  and  cost  savings.  The  testing  ap¬ 
proach  used  must  be  carefully  tailored  to 
the  type  of  system,  levels  of  modifications. 


and  the  amount  of  test  data  already  avail¬ 
able.  The  T&E  community  must  get  in¬ 
volved  early  in  the  process  so  that  all  test 
issues  are  adequately  addressed  and  timely 
comprehensive  evaluations  are  provided 
to  decision  authorities. 
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TESTING  THE  SPECIAL  CASES 


24.1  INTRODUCTION 

This  chapter  covers  the  special  factors  and 
alternative  test  strategies  the  tester  must 
consider  in  testing  dangerous  or  lethal 
weapons,  systems  that  involve  one-of-a- 
kind  or  limited  production,  advanced  con¬ 
cept  technology  demonstrations,  and  sys¬ 
tems  with  high-cost  and/or  special  secu¬ 
rity  considerations.  Examples  include 
chemical  and  laser  weapons;  ships;  space 
weapons;  and  missile  systems. 

24.2  TESTING  WITH  LIMITATIONS 

Certain  types  of  systems  cannot  be  tested 
using  relatively  standard  test  and  evalua¬ 
tion  (T&E)  approaches  for  reasons  such  as 
a  nonstandard  acquisition  strategy,  re¬ 
source  limitations,  cost,  safety  or  security 
constraints.  The  Test  and  Evaluation  Mas¬ 
ter  Plan  (TEMP)  must  contain  a  statement 
that  identifies  "those  factors  that  will  pre¬ 
clude  a  full  and  completely  realistic  opera¬ 
tional  test...(IOT&E  and  FOT&E),"  such  as 
inability  to  realistically  portray  the  entire 
threat,  limited  resources  or  locations,  safety 
and  system  maturity.  The  impact  of  these 
limitations  on  the  test's  critical  operational 
issues  must  also  be  addressed  in  the  TEMP. 

Nonstandard  acquisition  strategies  are  of¬ 
ten  used  for  one-of-a-kind  or  limited  pro¬ 
duction  systems.  Examples  of  these  include 
space  systems;  missiles;  and  ships.  For  one- 
of-a-kind  systems,  the  production  decision 
is  oftenmade  prior  to  system  design;  hence, 
testing  does  not  support  the  traditional 


decision  process.  In  limited  production  sys¬ 
tems,  there  are  often  no  prototypes  avail¬ 
able  for  test;  consequently,  the  tester  must 
develop  irmovative  test  strategies. 

The  tester  of  dangerous  or  lethal  systems, 
like  chemical  and  laser  weapons,  must  con¬ 
sider  various  safety,  health  and  medical 
factors  in  developing  test  plans,  such  as: 

(1 )  Provision  of  medical  facilities  for  pre- 
and  post-test  checkups  and  emergency 
treatment; 

(2)  Need  for  protective  gear  for  partici¬ 
pating/observer  personnel; 

(3)  Approval  of  the  test  plan  by  the  Sur¬ 
geon  General; 

(4)  Restrictions  in  selection  of  test  partici¬ 
pants  (e.g.,  medical  criteria  or  use  of  only 
volunteer  troops); 

(5)  Restricted  test  locations; 

(6)  Environmental  Impact  Statements. 

Also,  the  tester  must  allow  for  additional 
planning  time,  test  funds  and  test  resources 
to  accommodate  such  factors. 

24.2.1  Chemical  Weapons  Testing 

The  testing  of  chemical  weapons  poses 
unique  problems,  because  the  tester  cannot 


perform  actual  open-air  field  testing  with 
real  nerve  agents  or  other  toxic  chemi¬ 
cals.  Since  the  United  States  signed  and 
ratified  the  Geneva  Protocol  of  1925,  U.S. 
policy  has  been  that  the  United  States 
will  never  be  the  first  to  use  lethal  chemi¬ 
cal  weapons;  it  may,  however,  retaliate 
with  chemical  weapons  if  so  attacked.  In 
addition  to  the  health  and  safety  factors 
discussed  in  the  last  paragraph,  test  is¬ 
sues  the  chemical  weapons  tester  must 
address  include; 

(1)  All  possible  chemical  reactions  due  to 
variations  such  as  moisture,  temperature, 
pressure  and  contamination; 

(2)  Physical  behavior  of  the  chemical;  i.e., 
droplet  size,  dispersion  density  and  ground 
contamination  pattern  when  used  opera¬ 
tionally; 

(3)  Toxicity  of  the  chemical;  i.e.,  lethality 
and  duration  of  contamination  when  used 
operationally; 

(4)  Safety  of  the  chemical  weapon  during 
storage,  handling  and  delivery; 

(5)  Decontamination  process. 

Addressing  all  of  these  issues  requires  a 
combination  of  laboratory  toxic  chamber 
tests  and  open-air  field  testing.  The  latter 
must  be  performed  using  "simulants," 
which  are  substances  that  replicate  the 
physical  and  chemical  properties  of  the 
agent  but  with  no  toxicity. 

The  development  and  use  of  simulants  for 
testing  will  require  increased  attention  as 
more  chemical  weapons  are  developed. 
Chemical  agents  can  demonstrate  a  wide 
variety  of  effects  depending  on  such  factors 
as  moisture,  temperature  and  contamina¬ 
tion.  Consequently,  the  simulants  must  be 
able  to  replicate  all  possible  agent  reactions; 


it  is  likely  that  several  simulants  would 
have  to  be  used  in  a  test  to  produce  all 
predicted  agent  behaviors.  In  developing 
and  selecting  simulants,  the  tester  must 
thoroughly  understand  all  chemical  and 
physical  properties  and  possible  reactions 
of  the  agent. 

Studies  of  the  anticipated  reactions  can 
be  performed  in  toxic-chamber  tests  us¬ 
ing  the  real  agent.  Here,  factors  such  as 
changes  in  moisture,  temperature,  pres¬ 
sure  and  levels  of  impurity  can  be  con¬ 
trolled  to  assess  the  agent's  behavior.  But, 
the  tester  must  think  through  all  possible 
environmental  conditions  in  which  the 
weapon  could  operate  so  all  cases  can  be 
tested  in  the  laboratory  chamber  with  the 
real  agent.  For  example,  during  develop¬ 
ment  testing  of  the  BIGEYE  chemical 
weapon,  it  was  found  that  higher-than- 
expected  temperatures  due  to  aerody¬ 
namic  heating  caused  pressure  buildup 
in  the  bomb  body  that  resulted  in  the 
bomb  exploding.  This  caused  the  opera¬ 
tional  concept  for  the  BIGEYE  to  be 
changed  from  on-board  mixing  of  the 
two  chemicals  to  mixing  after  release  of 
the  bomb. 

T ests  to  confirm  toxicity  must  be  conducted 
using  simulants  in  the  actual  environment. 
Since  the  agent's  toxicity  is  dependent  on 
factors  such  as  droplet  size,  dispersion  den¬ 
sity,  ground  contamination  pattern  and 
degradation  rate,  a  simulant  that  behaves 
as  the  agent  does  must  be  used  in  actual 
field  testing.  Agent  toxicity  is  determined 
in  the  lab. 

The  Services  publish  a  variety  of  technical 
documents  on  specific  chemical  test  pro¬ 
cedures.  Documents  such  as  the  U.S.  Army 
Test  and  Evaluation  Command  (TECOM) 
Pamphlet  310-4,  a  bibliography  that  in¬ 
cludes  numerous  reports  on  chemical  test¬ 
ing  issues  and  procedures,  can  be  con- 
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suited  for  specific  documentation  on 
chemical  testing. 

24.2.3  Laser  Weapons  Testing 

Many  new  weapon  systems  are  being  de¬ 
signed  with  embedded  laser  range  finders 
and  laser  designators.  Because  of  the  dan¬ 
ger  to  the  human  eye  posed  by  lasers,  the 
tester  must  adhere  to  special  safety  require¬ 
ments  and  utilize  special  locations  during 
T&E.  For  instance,  the  only  Army  installa¬ 
tion  in  the  continental  United  States  per¬ 
mitting  free-play  airborne  laser  testing  is 
Fort  Hunter-Liggett,  CA.  During  tests  in¬ 
volving  lasers,  the  airspace  must  be  re¬ 
stricted;  and  guards  must  be  posted  to  pre¬ 
vent  anyone  from  accidentally  venturing 
into  the  area.  A  potential  solution  to  the 
safety  issue  is  to  develop  and  use  an  "eye- 
safe"  laser  for  testing.  The  tester  must  en¬ 
sure  that  eye-safe  lasers  produce  the  same 
laser  energy  as  the  real  laser  system. 

Another  concern  of  the  laser  energy  weap¬ 
ons  tester  is  the  accurate  determination  of 
laser  energy  level  and  location  on  the  tar¬ 
get.  Measurements  of  the  laser  energy  on 
the  target  are  usually  conducted  in  the 
laboratory  as  part  of  development  test  (DT) . 
In  the  field,  video  cameras  are  often  used  to 
verify  that  the  laser  designator  did  indeed 
illuminate  the  target.  Such  determinations 
are  important  when  the  tester  is  trying  to 
attribute  weapon  performance  to  behavior 
of  the  laser,  behavior  of  the  guidance  sys¬ 
tem,  or  some  other  factor. 

A  bibliography  of  Army  test  procedures, 
TECOM  Pamphlet  310-4,  lists  several  docu¬ 
ments  that  cover  the  special  issues  associ¬ 
ated  with  laser  testing. 

24.3  SPACE-SYSTEM  TESTING 

From  a  historical  perspective,  space-sys¬ 
tem  acquisition  has  posed  several  unique 
problems  to  the  test  process  (especially 


the  operational  test  process)  that  gener¬ 
ally  fall  into  four  categories:  limited  quan¬ 
tities/high  cost,  "block  upgrade"  approach 
to  acquisition,  operating  environment 
(peacetime  and  wartime),  and  test  envi¬ 
ronment. 

(1)  Limited  quantities /high  cost  -  Space 
systems  have  traditionally  involved  the 
acquisition  of  relatively  few  (historically, 
less  than  20)  systems  at  extremely  "high 
per-unit  costs"  (in  comparison  with  more 
traditional  military  systems).  The  high  per- 
unit  costs  are  driven  by  a  combination  of 
high  transportation  costs  (launch  to  orbit), 
high  life-cycle  reliability  requirements  and 
associated  costs  because  of  the  lack  of  an 
"on-orbit"  maintenance  capability  and  the 
high  costs  associated  with  "leading  edge" 
technologies  that  tend  to  be  a  part  of  space¬ 
craft  design.  From  a  test  perspective,  this 
serves  to  drive  space-system  acquisition 
strategy  into  the  "nonstandard"  approach 
addressed  below.  The  problem  is  com¬ 
pounded  by  the  'block  upgrade"  approach 
to  acquisition. 

(2)  Block  upgrade  approach  to  acquisi¬ 
tion  -  Due  to  the  "limited  buy"  and  "high 
per-unit  cost"  nature  of  spacecraft  acqui¬ 
sition,  these  systems  tend  to  be  procured 
using  a  "block  upgrade"  acquisition  strat¬ 
egy.  Under  this  concept,  "the  decision  to 
deploy"  is  often  made  at  the  front  end  of 
the  acquisition  cycle;  and  the  first  proto¬ 
type  to  be  placed  in  orbit  becomes  the  first 
operational  asset.  As  early  and  follow-on 
systems  undergo  ground  and  on-orbit  test¬ 
ing  (either  development  test  and  evalua¬ 
tion  (DT&E)  or  operational  test  and  evalu¬ 
ation  (OT &E)),  discrepancies  are  corrected 
by  "block  changes"  to  the  next  system  in 
the  pipeline.  This  approach  to  acquisition 
can  perturb  the  test  process  as  the  tester 
may  have  no  formal  milestone  decisions 
to  test  toward.  The  focus  must  change 
toward  being  able  to  influence  the  design 
of  (and  block  changes  to)  systems  further 
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downstream  in  the  pipeline.  As  the  first 
"on-orbit"  asset  usually  becomes  the  first 
operational  asset,  pressure  is  created  from 
the  operational  community  to  expedite  (and 
sometimes  limit)  testing  so  a  limited  opera¬ 
tional  capability  can  be  declared  and  the 
system  can  begin  fulfilling  mission  require¬ 
ments.  Once  the  asset  "goes  operational," 
any  use  of  it  for  testing  must  compete  with 
operational  mission  needs  —  a  situation 
potentially  placing  the  tester  in  a  position 
of  relatively  low  priority.  Recognition  of 
these  realities  and  careful  "early-on"  test 
planning  can  overcome  many  of  these  prob¬ 
lems,  but  the  tester  needs  to  be  involved 
and  ready  much  earlier  in  the  cycle  than 
with  traditional  systems. 

(3)  Operating  environment  (peacetime 
and  wartime)  -  Most  currently  deployed 
space  systems  and  near-term  future  space 
systems  operate  in  the  military  support 
arena  such  as  tactical  warning/attack  as¬ 
sessment,  communications,  navigation, 
weather  and  intelligence;  and  their  day-to- 
day  peacetime  operating  environment  is 
not  much  different  from  the  wartime  oper¬ 
ating  environment  except  for  activity  level 
(i.e.,  message  throughput,  more  objects  to 
track/see,  etc.).  Historically,  space  has  been 
a  relatively  benign  battlefield  environment 
because  of  technology  limitations  in  the 
capability  of  potential  adversaries  to  reach 
into  space  with  weapons.  But  this  is  no 
longer  valid.  This  combination  of  support- 
type  missions  and  a  battlefield  environ¬ 
ment  that  is  not  much  different  from  the 
peacetime  environment  has  played  a  defi¬ 
nite  role  in  allowing  systems  to  reach  lim¬ 
ited  operational  capability  without  as  much 
dedicated  prototype  system-level  testing 
as  seen  on  other  systems.  This  situation  is 
changing  with  the  advent  of  concepts  like 
the  Ballistic  Missile  Defense  system  where 
actual  weapons  systems  (impact  anti-satel¬ 
lite  and  laser)  may  be  in  operation,  and 
day-to-day  peacetime  operations  will  not 


mirror  the  anticipated  battlefield  environ¬ 
ment  as  closely.  Likewise,  the  elevation  of 
the  battlefield  into  space  and  the  advanc¬ 
ing  technologies  that  allow  potential  ad¬ 
versaries  to  reach  into  space  is  changing  the 
thrust  of  how  space  systems  need  to  be 
tested  in  space.  The  Department  of  Defense 
(DoD)  should  anticipate  an  increased  need 
for  dedicated  on-orbit  testing  on  a  type  of 
space  range  where  the  battlefield  environ¬ 
ment  will  be  replicated  can  be  anticipated 
—  a  situation  similar  to  the  dedicated  test¬ 
ing  done  today  on  test  ranges  with  Army, 
Navy  and  Air  Force  weapons. 

(4)  Test  environment  -  The  location  of 
space  assets  in  "remote"  orbits  also  com¬ 
pounds  the  test  problem.  Space  systems  do 
not  have  the  ready  access  (as  with  ground 
or  aircraft  systems)  to  correct  deficiencies 
identified  during  testing.  This  situation  has 
driven  the  main  thrust  of  testing  into  the 
"prelaunch"  ground  simulation  environ¬ 
ment  where  discrepancies  can  be  corrected 
before  the  system  becomes  inaccessible. 
However,  as  mentioned  previously,  when 
space-system  missions  change  from  a  war- 
support  focus  to  a  war-fighting  focus  and 
the  number  of  systems  required  to  do  the 
mission  increases  from  the  "high  reliabil¬ 
ity/limited  number"  mode  to  a  more  tradi¬ 
tional  "fairly  large  number  buy"  mode, 
future  space-system  testing  could  be  ex¬ 
pected  to  become  more  like  the  testing  as¬ 
sociated  with  current  ground,  sea  and  air 
systems.  From  a  test  perspective,  this  could 
also  create  unique  "test  technology"  re¬ 
quirements;  i.e.,  with  these  systems  we  will 
have  to  bring  the  test  range  to  the  operating 
system  as  opposed  to  bringing  the  system 
to  the  range.  Also,  because  the  space  envi¬ 
ronment  tends  to  be  "visible  to  the  world" 
(others  can  observe  our  tests  as  readily  as 
we  can),  unique  test  operations  security 
methodologies  may  be  required  to  allow  us 
to  achieve  test  realism  without  giving  away 
system  vulnerabilities. 
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In  summary,  current  and  near-term  future 
space  systems  have  unique  test  method¬ 
ologies.  However,  in  the  future,  space  op¬ 
erations  might  entail  development/deploy¬ 
ment  of  weapon  platforms  on  orbit  with 
lower  design-life  reliability  (because  of 
cost);  and  day-to-day  peacetime  operations 
will  not  mirror  the  wartime  environment. 
Thus,  space-system  testing  requirements 
may  begin  to  more  closely  parallel  those  of 
traditional  weapon  systems. 

24.4  OPERATIONS  SECURITY 
AND  T&E 

Operations  security  (OPSEC)  issues  must 
be  considered  in  all  test  planning.  Security 
regulations  and  contracting  documents  re¬ 
quire  the  protection  of  "sensitive  design 
information  and  test  data"  throughout  the 
acquisition  cycle  by: 

(1)  Protecting  sensitive  technology; 

(2)  Eliminating  nonsecure  transmittal 
data  on  and  from  test  ranges; 

(3)  Providing  secure  communications 
linking  DoD  agencies  to  each  other  and  to 
their  contractors. 

Such  protection  is  obviously  costly  and 
will  require  additional  planning  time,  test 
resources  and  test  constraints.  The  test  plan¬ 
ner  must  determine  all  possible  ways  in 
which  the  system  could  be  susceptible  to 
hostile  exploitation  during  testing.  For  ex¬ 
ample,  announcement  of  test  schedule  and 
location  could  allow  monitoring  by  unau¬ 
thorized  persons.  Knowledge  of  the  loca¬ 
tions  of  systems  and  instrumentation  or 
test  concepts  could  reveal  classified  system 
capabilities  or  military  concepts.  Compila¬ 
tions  of  unclassified  data  could,  as  a  whole. 


reveal  classified  information  as  could  sur¬ 
veillance  (electronic  or  photographic)  of 
test  activities  or  interception  of  unencrypted 
transmissions.  TheT&E  regulations  of  each 
Service  require  an  operational  security  plan 
for  a  test.  A  detailed  list  of  questions  the  test 
planner  can  use  to  identify  the  potential 
threat  of  exploitation  is  provided  in  APR 
55-43. 

24.5  Advanced  Technology  Concept 
Demonstrations 

Systems  with  potential  utility  for  the  user 
and  having  relatively  mature  technology 
may  be  evaluated  by  a  user  in  an  opera¬ 
tional  field  environment.  These  programs 
are  not  an  acquisition  program  and  there¬ 
fore  are  not  subject  to  the  normal  acquisi¬ 
tion  T&E  processes.  A  favorable  evaluation 
may  result  in  the  decision  to  acquire  addi¬ 
tional  systems  for  Service  use,  bypassing  a 
number  of  the  normal  acquisition  phases. 
The  Services  have  been  using  their  opera¬ 
tional  test  agencies  to  assist  the  field  com¬ 
manders  in  structuring  an  evaluation  pro¬ 
cess  which  would  provide  the  documented 
data  necessary  for  an  informed  acquisition 
decision. 

24.6  SUMMARY 

All  weapon  systems  tests  are  limited  to 
some  degree,  but  certain  systems  face  ma¬ 
jor  limitations  that  could  preclude  a  com¬ 
prehensive  and  realistic  evaluation.  The 
test  planners  of  these  special  systems  must 
allow  additional  planning  time,  budget  for 
extra  test  resources  and  devise  alternative 
test  strategies  to  work  around  testing  limi¬ 
tations  caused  by  such  factors  as  security 
restrictions,  resource  availability,  environ¬ 
mental  safety  factors  and  nonstandard  ac¬ 
quisition  strategies. 
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T&E  WEAPON  SYSTEMS  TYPES 


25.1  INTRODUCTION 

This  chapter  offers  guidance  to  Depart¬ 
ment  of  Defense  personnel  who  plan, 
monitor  and  execute  test  and  evaluation 
(T&E).  Checklists  for  the  chapter  were  ob¬ 
tained  from  the  Defense  Science  Board 
(DSB)  Study,  Report  of  Task  Force  on  Test 
and  Evaluation,  dated  April  2,  1974.  This 
excellent  study  is  highly  regarded  in  the 
T&E  community  but  has  become  dated; 
consequently,  the  Defense  Systems  Man¬ 
agement  College  decided  to  update  the 
study  findings  and  include  those  findings 
and  summary  checklists  in  this  manage¬ 
ment  guide. 

25.2  SPECIFIC  WEAPON  SYSTEMS 
TESTING  CHECKLIST 

The  DSB  report  is  the  result  of  the  study  of 
past  major  weapon  systems  acquisitions.  It 
was  hoped  that  this  study  would  enhance 
the  testing  community's  understanding  of 
the  role  that  T&E  has  had  in  identifying 
system  problems  during  the  acquisition 
process.  In  the  foreword  of  the  DSB  study, 
the  authors  made  this  statement  about  in¬ 
cluding  the  obvious  testing  activity  in  their 
checklist: 

The  T&E  expert  in  reading  this  volume  will 
find  many  precepts  which  will  strike  him 
as  of  this  type.  These  items  are  included 
because  examples  were  found  where  even 
the  obvious  has  been  neglected,  not  be¬ 
cause  of  incompetence  or  lack  of  personal 
dedication  by  the  people  in  charge  of  the 
program,  but  because  of  financial  and 


temporal  pressures  which  forced  compe¬ 
tent  managers  to  compromise  on  their  prin¬ 
ciples.  It  is  hoped  that  the  inclusion  of  the 
obvious  will  prevent  repetition  of  the  seri¬ 
ous  errors  which  have  been  made  in  the 
past  when  such  political,  economical  and 
temporal  pressures  have  forced  project 
managers  to  depart  from  the  rules  of  sound 
engineering  practices....In  the  long  run,  tak¬ 
ing  short  cuts  during  T&E  to  save  time  and 
money  will  result  in  significant  increases  in 
the  overall  costs  of  the  programs  and  in  a 
delay  of  delivery  of  the  corresponding 
weapon  systems  to  combatant  forces. 

25.2.1  Aircraft  Systems 

25.2.1.1  Concept  Exploration 
Phase 

•  Test  Program/Total  Costs.  Prior  to  Mile¬ 
stone  I,  all  phases  of  the  aircraft  test  pro¬ 
gram  should  be  considered  so  the  total 
costs  and  the  development  schedules  in¬ 
clude  consideration  of  all  likely  activities  in 
the  overall  program. 

•  Test  Facilities  and  Instrumentation.  Prior 
to  Milestone  I,  the  test  facilities  and  instru¬ 
mentation  requirements  to  conduct  tests 
should  be  generally  identified  along  with  a 
tentative  schedule  of  test  activities. 

•  Test  Resources  and  Failures.  Ensure  that 
there  are  adequate  funds,  reasonable 
amounts  of  time,  and  acceptable  num¬ 
bers  of  aircraft  planned  for  the  various 
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test  program  phases,  and  that  provisions 
are  made  for  the  occurrence  of  failures. 

'  •  System  Interfaces.  Consider  all  aircraft 
system  interfaces,  their  test  requirements, 
and  probable  costs  at  the  outset  of  the  Con¬ 
cept  Exploration  Phase. 

•  Major  Weapon  Subsystems.  If  the  aircraft 
system  relies  on  the  successful  develop¬ 
ment  of  a  specific  and  separately-funded 
major  weapon  (such  as  a  gun  or  missile)  in 
order  to  accomplish  its  primary  mission, 
this  major  subsystem  should  be  developed 
and  tested  concurrently  with,  or  prior  to, 
the  aircraft. 

•  Propulsion  System.  If  the  aircraft  pro¬ 
gram  is  paced  by  the  propulsion  system 
development,  an  early  advanced-develop¬ 
ment  project  for  the  propulsion  may  be 
appropriate  for  a  new  concept. 

•  Operational  Scenario.  A  conceptual  op¬ 
erational  scenario  for  operation  and  use  of 
the  aircraft  should  be  developed  so  that 
general  test  plans  can  be  designed.  This 
should  include  purpose,  roles  and  mis¬ 
sions,  threats,  operating  environments,  lo¬ 
gistics  and  maintenance  and  basing  char¬ 
acteristics.  The  potential  range  of  values  on 
these  aspects  should  be  stated. 

•  Evaluation  Criteria.  Develop  evaluation 
criteria  to  be  used  for  selecting  the  final 
aircraft  system  design. 

•  Untried  Elements.  The  aircraft  develop¬ 
ment  program  should  include  conclusive 
testing  to  eliminate  uncertainties  of  the 
untried  elements. 

•  Brassboard  Avionics  Tests.  The  use  of 
brassboard  or  modified  existing  hardware 
to  "prove"  the  concept  will  work  should  be 
seriously  scrutinized  to  ensure  that  the  dem¬ 
onstrations  and  tests  are  applicable. 


•  Nuclear  Weapons  Effects.  The  subject  of 
nuclear  weapons  effects  should  be  ad¬ 
dressed  in  the  test  concept  for  all  aircraft 
weapons  systems  where  operational  suit¬ 
ability  dictates  that  survivable  exposure  to 
nuclear  weapons  effects  is  a  requirement. 

25.2.1.2  Program  Definition  and 
Risk  Reduction  Phase 

•  By  the  end  of  the  phase,  T&E  plans  and 
test  criteria  should  be  established  so  there 
is  no  question  on  what  constitutes  a  suc¬ 
cessful  test  and  what  performance  is  re¬ 
quired. 

•  Milestones  and  Goals.  Ensure  an  inte¬ 
grated  system  test  plan  that  pre-establishes 
milestones  and  goals  for  easy  measure¬ 
ment  of  program  progress  at  a  later  time. 

•  Operating  Concept  and  Environment.  The 
operational  concept  and  the  environments 
in  which  the  aircraft  will  be  expected  to 
operate  and  be  tested  in  OT&E  should  be 
specified. 

•  Test  Program  Building  Blocks.  In  the 
PDRR  Phase,  demonstrate  that  high-risk 
technology  is  in  hand.  In  planning  the  sys¬ 
tem  level  test  program,  ensure  that  compo¬ 
nents  and  subsystems  are  adequately  quali¬ 
fied  for  incorporation  into  the  system  tests. 

•  Technology  Concepts.  Each  concept  to  be 
used  in  the  aircraft  system  (e.g.,  aerody¬ 
namics,  structures,  propulsion)  should  be 
identified  and  coded  according  to  prior 
application,  before  future  research.  Tests 
for  each  concept  should  be  specified  with 
the  effect  of  failure  identified. 

•  DT&E/OT&EP/fln.  The  aircraft  DT&E/ 
OT&E  test  plan  should  be  reviewed  to 
ensure  it  includes  ground  and  flight  tests 
necessary  to  safely  and  effectively  develop 
the  system. 
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•  Test  Failures.  The  T&E  plans  should 
be  made  assuming  there  will  be  failures; 
they  are  inevitable. 

•  Multi-Service  Testing.  When  a  new 
aircraft  development  program  requires 
multi-Service  testing  during  OT&E  and 
prior  to  Milestone  II,  the  test  plan  should 
include  the  types  of  tests  and  resources 
required  from  other  activities  and  Ser¬ 
vices. 

•  Traceability.  The  aircraft  development 
and  test  program  should  be  designed  and 
scheduled  so  if  trouble  arises,  its  source 
can  be  traced  back  through  the  lab  tests 
and  the  analytical  studies. 

•  Competitive  Prototype  Tests.  When  a 
competitive  prototype  test  program  us¬ 
ing  test  and  operational  crews  is  em¬ 
ployed,  the  aircraft  should  be  compared 
on  the  basis  of  the  performance  of  critical 
missions. 

•  Prototype  Similarity  to  Development  and 
Production  Aircraft.  A  firm  determination 
should  be  made  of  the  degree  of  similar¬ 
ity  of  the  winning  prototype  (in  a  com¬ 
petitive  prototype  program)  to  the  engi¬ 
neering  development  model  and  produc¬ 
tion  aircraft.  Thus,  test  results  that  are 
derived  from  the  prototype  in  the  interim 
period  prior  to  availability  of  the  engi¬ 
neering  development  model  aircraft  can 
be  utilized  effectively. 

•  Prototype  Tests.  The  prototype  air¬ 
craft  test  data  should  be  used  to  deter¬ 
mine  where  emphasis  should  be  placed 
in  the  engineering  development  program. 

•  Inlet/Engine/Nozzle  Match.  The  air¬ 
craft  test  program  should  provide  for  an 
early  and  adequate  inlet/ engine/ nozzle 
match  through  a  well-planned  test  pro¬ 
gram,  and  there  should  be  time  program¬ 
ming  for  corrections. 


•  Subsystem  Tests.  There  should  be  a 
balanced  program  for  the  aircraft  sub¬ 
system  tests. 

•  Propulsion  System.  If  the  aircraft  is 
paced  by  the  propulsion  systems  devel¬ 
opment,  an  early  advanced-development 
project  for  the  propulsion  may  be  appro¬ 
priate  for  a  new  concept. 

•  Electromagnetic  Interface  (EMI)  Test¬ 
ing.  Full-scale  aircraft  systems  tests  in  an 
anechoic  chamber  are  desirable  for  some 
aircraft. 

•  Parts  Interchange.  Early  plans  should 
provide  for  tests  where  theoretically  iden¬ 
tical  parts,  particularly  in  avionics,  are 
interchanged  to  ensure  that  the  aircraft 
systems  can  be  maintained  in  readiness. 

•  Human  Factors  Demonstration.  Ensure 
adequate  demonstration  of  human  fac¬ 
tors  is  considered  in  test  planning. 

•  Logistics  T&E.  Adequate  resources 
should  be  scheduled  for  the  aircraft  logis¬ 
tics  system  T&E  and  a  positive  program 
should  exist  for  the  utilization  of  this 
information  at  the  time  of  OT&E. 

•  User  Participation.  It  is  imperative  that 
the  operational  command  actively  par¬ 
ticipate  in  the  DT&E  Phase  to  ensure  that 
user  needs  are  represented  in  the  devel¬ 
opment  of  the  system. 

25.2.1.3  Engineering  and 
Manufacturing  Development 
Phase 

•  Test  Design.  Test  programs  should  be 
designed  to  have  a  high  probability  of 
early  identification  of  major  deficiencies 
during  the  DT&E  and  lOT&E. 

•  Data  for  Alternate  Scenarios.  By  careful 
attention  to  testing  techniques,  maximize 
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the  utility  of  the  test  data  gathered;  aircraft 
instrumentation;  range  instrumentation; 
and  data  collection,  reduction  and  storage. 

•  Test  Milestones.  Development  programs 
should  be  built  around  testing  milestones, 
not  calendar  dates. 

•  Production  Engineering  Influence  on  R&D 
Hardware.  Encourage  that  production  phi¬ 
losophy  and  production  techniques  be 
brought  to  the  maximum  practicable  ex¬ 
tent  into  an  early  phase  of  the  design  pro¬ 
cess  for  R&D  hardware. 

•  Running  Evaluation  of  Tests.  Ensure  that 
running  evaluations  of  tests  are  conducted. 
If  it  becomes  clear  that  test  objectives  are 
unattainable  or  additional  samples  will  not 
change  the  test  outcome,  ensure  that  proce¬ 
dures  are  established  for  terminating  the 
test. 

•  Simulation.  Analysis  and  simulation 
should  be  conducted,  where  practicable, 
before  each  phase  of  development  flight 
testing. 

•  Avionics  Mock-up.  Encourage  use  of  a 
complete  avionics  system  installed  in  a 
mock-up  of  the  appropriate  section  or  sec¬ 
tions  of  the  aircraft. 

•  Escape  Systems  Testing.  Ensure  the  air¬ 
crew  escape  system  is  thoroughly  tested 
with  particular  attention  to  redundant  fea¬ 
tures,  such  as  pyrotechnic  firing  channels. 

•  Structural  Testing.  Ensure  that  fatigue 
testing  is  conducted  on  early  production 
airframes.  Airframe  production  should  be 
held  to  a  low  rate  until  satisfactory  progress 
is  shown  in  these  tests. 

•Gun  Firing  Tests.  All  forms  of  ordnance, 
especially  those  that  create  gases,  must  be 
fired  from  the  aircraft  for  external  effects 
(blast  and  debris),  internal  effects  (shock) 


and  effects  on  the  propulsion  (inlet  compo¬ 
sition  or  distribution). 

•  Post-Stall  Characteristics.  Special  atten¬ 
tion  is  warranted  on  the  post-stall  test  plans 
for  DT&E  and  OT&E. 

•  Subsystem  Performance  History.  During 
DT&E  and  lOT&E  of  aircraft,  ensure  that  a 
performance  history  of  each  aircraft  sub¬ 
system  is  kept. 

•  Flight  Deficiency  Reporting.  Composi¬ 
tion  of  flight  deficiencies  reporting  by  air¬ 
crews,  particularly  those  pertaining  to  avi¬ 
onics,  should  be  given  special  attention. 

•  Crew  Limitations.  Ensure  aircrew  limi¬ 
tations  are  included  in  the  tests. 

•  Use  of  Operational  Personnel.  Recom¬ 
mend  experienced  operational  personnel 
help  in  establishing  measures  of  effective¬ 
ness  and  in  other  operational  test  planning. 
In  conducting  OT&E,  use  typical  opera¬ 
tional  aircrews  and  support  personnel. 

•  Role  of  the  User.  Ensure  that  users  par¬ 
ticipate  in  the  T&E  phase  so  their  needs  are 
represented  in  the  development  of  the  sys¬ 
tem  concept  and  hardware. 

•  Crew  Fatigue  and  System  Effectiveness.  In 
attack  aircraft  operational  testing  and  par¬ 
ticularly  in  attack  helicopter  tests  where 
vibration  is  a  fatiguing  factor,  ascertain 
that  the  tests  include  a  measure  of  degrada¬ 
tion  over  time. 

•  Time  Constraints  on  Crews.  Detailed  op¬ 
erational  test  plans  should  be  evaluated  to 
determine  that  the  test-imposed  conditions 
on  the  crew  do  not  invalidate  the  applica¬ 
bility  of  the  collected  data. 

•  Maintenance  and  Training  Publications. 
The  aircraft  development  program  should 
provide  for  concurrent  training  of  crews 
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and  preparation  of  draft  technical  manu¬ 
als  to  be  used  by  lOT&E  maintenance  and 
operating  crews. 

•  Research  and  Development  (R&D) 
Completion  Prior  to  lOT&E.  The  testing 
plans  should  ensure  that,  before  an  air¬ 
craft  system  is  subjected  to  lOT&E,  the 
subsystems  essential  to  the  basic  mission 
have  completed  R&D. 

•  Complete  Basic  DT&E  before  Starting 
OT&E.  Before  the  weapon  system  is  sub¬ 
jected  to  lOT&E,  all  critical  subsystems 
should  have  completed  basic  DT&E  and 
significant  problems  should  be  solved. 

•  Realism  in  Testing.  Ascertain  that  final 
DT&E  system  tests  and  lOT&E  flight  tests 
are  representative  of  operational  condi¬ 
tions. 

•  Test  All  Profiles  and  Modes.  Tests  should 
be  conducted  to  evaluate  all  planned  op¬ 
erational  flight  profiles  and  all  primary 
and  backup,  degraded  operating  modes. 

•  Update  of  Operational  Test  Plans.  En¬ 
sure  that  operational  test  plans  are  re¬ 
viewed  and  updated,  as  needed,  to  make 
them  relevant  to  evolving  concepts. 

•  Conduct  OT&E  Early.  Ensure  that  op¬ 
erational  suitability  tests  are  planned  to 
attempt  to  identify  operational  deficien¬ 
cies  of  new  systems  quickly  so  fixes  can 
be  developed  and  tested  before  large- 
scale  production. 

•  Missile  Launch  Tests.  Review  the  final 
position  fix  planned  before  launching  in- 
ertial-guided  air-to-surface  missiles. 

•  Mission  Completion  Success  Probabil¬ 
ity.  Mission  completion  success  probabil¬ 
ity  factors  should  be  used  to  measure 
progress  in  the  aircraft  test  program. 


25.2.1.4  Production,  Fielding 
Deployment  and  Operational 
Support  Phase 

•  Operational  Test  Realism.  Assure  FOTE 
is  conducted  under  realistic  conditions. 

•  Design  FOT&E  for  Less-Than-Optimal 
Condition.  Structure  the  FOT&E  logistical 
support  for  simulated  combat  conditions. 

•  New  Threat.  Be  alert  to  the  need  to 
extend  the  FOT&E  if  a  new  threat  shows 
up. 

•  Certification  of  Ordnance.  Ensure  that 
ordnance  to  be  delivered  by  an  aircraft  is 
certified  for  the  aircraft. 

•  Inadvertent  Influence  of  Test.  The 
FOT&E  plans  should  provide  measures 
for  ensuring  that  actions  by  observers 
and  umpires  do  not  unwittingly  influ¬ 
ence  trial  outcome. 

•  Deficiencies  Discovered  In-Service.  Be 
aware  that  in-Service  operations  of  an 
aircraft  system  will  surface  deficiencies 
which  extensive  FOT&E  probably  would 
not  uncover. 

•  Lead  the  Fleet.  Accelerated  Service  test 
of  a  small  quantity  of  early  production 
aircraft  is  advisable  and  during  FOT&E 
thereafter. 

25.2.2  Missile  Systems 

25.2.2.1  Concept 
Exploration  Phase 

•  Weapon  System  Interfaces.  Consider 
significant  weapon  system  interfaces, 
their  test  requirements  and  probable  costs 
at  the  outset  of  the  Concept  Exploration 
Phase.  Ensure  that  the  program  plan  as¬ 
sembled  before  Milestone  I  includes  and 
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understanding  of  the  basic  test  criteria  and 
broad  test  plans  for  the  whole  program. 

•  Number  of  Test  Missiles.  Ensure  that 
there  is  sufficient  time  and  a  sufficient  num¬ 
ber  of  test  articles  to  support  the  program 
through  its  various  phases.  Compare  the 
program  requirements  with  past  missile 
programs  of  generic  similarity.  If  there  is 
substantial  difference,  then  adequate  justi¬ 
fication  should  be  provided.  The  DT&E 
period  on  many  programs  has  had  to  be 
extended  as  much  as  50  percent. 

•  Test  and  Evaluation  Gap.  A  T&E  gap  has 
been  experienced  in  some  missile  programs 
between  the  time  when  testing  with  R&D 
hardware  was  completed  and  the  time  when 
follow-on  operational  suitability  testing  was 
initiated  with  production  hardware. 

•  Feasibility  Tests.  Ensure  experimental 
test  evidence  is  available  to  indicate  the 
feasibility  of  the  concept  and  the  availabil¬ 
ity  of  the  technology  for  the  system  devel¬ 
opment. 

•  Evaluation  of  Concept  and  Prototype  Tests. 
Results  of  tests  conducted  during  the  Con¬ 
cept  Exploration  and  the  PDRR  Phases, 
wfuch  most  likely  have  been  conducted  as 
avionics  brassboard,  breadboard  or  modi¬ 
fied  existing  hardware,  should  be  evalu¬ 
ated  with  special  attention. 

•  Multi-Service  TestingPlans.  Whena  new 
missile  development  program  requires 
multi-Service  testing  during  OT&E,  the 
TEMP  should  include  the  type  of  tests  and 
resources  required  from  other  activities  and 
Services. 

•  Test  Facilities  and  Instrumentation  Re¬ 
quirements.  Before  Milestone  I,  the  test  fa¬ 
cilities  and  instrumentation  requirements 
to  conduct  tests  should  be  generally  identi¬ 
fied  along  with  a  tentative  schedule  of  test 
activities. 


25.2.2.2  Program  Definition 
and  Risk  Reduction  Phase 

•  Establish  Test  Criteria.  By  the  end  of  the 
PDRR  phase,  test  criteria  should  be  estab¬ 
lished  so  there  is  no  question  on  what  consti¬ 
tutes  a  successful  test  and  what  perfor¬ 
mance  is  expected. 

•  Human  Factors.  Ensure  that  the  TEMP 
includes  adequate  demonstration  of  hu¬ 
man  factors  considerations. 

•  Instrumentation  Diagnostic  Capability  and 
Compatibility.  Instrumentationdesign,  with 
adequate  diagnostic  capability  and  com¬ 
patibility  in  DT&E  and  lOT&E  phases,  is 
essential. 

•  Provisions  for  Test  Failures.  The  DT&E 
and  OT&E  plans  should  include  provisions 
for  the  occurrence  of  failures. 

•  Integrated  Test  Plan.  Ensure  develop¬ 
ment  of  an  integrated  system  test  plan  that 
pre-establishes  milestones  and  goals  for 
easy  measurement  of  program  progress  at 
a  later  time. 

•  Test  and  Evaluation  Requirements.  En¬ 
sure  that  the  T&E  program  requirements 
are  firm  before  approving  an  R&D  test 
program.  Many  missile  programs  have  suf¬ 
fered  severe  cost  impacts  as  a  result  of  this 
deficiency.  The  test  plan  must  include  pro¬ 
visions  to  adequately  test  those  portions  of 
the  operational  envelope  that  stress  the 
system  including  backup  and  degraded 
operational  modes. 

•  Personnel  Training  Plans.  Ensure  that 
adequate  training  and  certification  plans 
for  test  personnel  have  been  developed. 

•  Test  and  Engineering  Reporting  Format. 
Include  a  T&E  reporting  format  in  the 
program  plan.  Attention  must  be  given  to 
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the  reporting  format  in  order  to  provide  a 
consistent  basis  for  T&E  throughout  the 
program  life  cycle. 

•  Program-to-Program  Cross  Talk.  Encour¬ 
age  program-to-program  T&E  cross  talk. 
Test  and  evaluation  problems  and  their 
solutions,  as  one  program,  provide  a  valu¬ 
able  index  of  lessons  learned  and  tech¬ 
niques  for  problem  resolution  on  other  pro¬ 
grams. 

•  Status  of  T&E  Offices.  Ensure  that  T&E 
offices  reporting  to  the  program  manager 
or  director  have  the  same  stature  as  other 
major  elements.  It  is  important  that  the 
T&E  component  of  the  system  program 
office  has  organizational  status  and  au¬ 
thority  equal  to  configuration  management, 
program  control,  system  engineering,  etc. 

•  Measurement  of  Actual  Environments. 
Thorough  measurements  should  be  made 
to  define  and  understand  the  actual  envi¬ 
ronment  in  which  the  system  components 
must  live  during  the  captive,  launch  and 
in-flight  phases. 

•  Thoroughness  of  Laboratory  Testing.  Sig¬ 
nificant  time  and  money  will  be  saved  if 
each  component,  each  subsystem,  and  the 
full  system  are  all  tested  as  thoroughly  as 
possible  in  the  laboratory. 

•  ContractForm.  The  contract  form  can  be 
extremely  important  to  the  T&E  aspects.  In 
one  program,  the  contract  gave  the  contrac¬ 
tor  full  authority  to  determine  the  number 
of  test  missiles;  and  in  another,  the  contract 
incentive  resulted  in  the  contractor  concen¬ 
trating  tests  on  one  optimum  profile  to 
satisfy  the  incentive  instead  of  developing 
the  performance  throughout  important  ar¬ 
eas  of  the  envelope. 

•  Participation  of  Operational  Command.  It 
is  imperative  that  the  operational  command 


actively  participate  in  the  DT&E  phase  to 
ensure  that  user  needs  are  represented  in 
the  development  of  the  system. 

25.2.2.3  Engineering 
and  Manufacturing 
Development  Phase 

•  Production  Philosophy  and  Techniques. 
Encourage  that  production  philosophy  and 
production  techniques  be  brought,  to  the 
maximum  practicable  extent,  into  an  early 
phase  of  the  design  process  for  R&D  hard¬ 
ware.  There  are  many  missile  programs  in 
which  the  components  were  not  qualified 
until  the  missile  was  well  into  production. 

•  Operational  Flight  Profiles.  Tests  should 
be  conducted  to  evaluate  all  planned  op¬ 
erational  flight  profiles  and  all  primary  and 
backup  degraded  operating  modes. 

•  Failure  Isolation  and  Responsive  Action. 
Does  the  system  test  plan  provide  for  ad¬ 
equate  instrumentation  so  missile  failures 
can  be  isolated  and  fixed  before  the  next 
flight? 

•  Responsive  Actions  for  Test  Failures.  En¬ 
courage  a  closed-loop  reporting  and  reso¬ 
lution  process,  which  ensures  that  each  test 
failure  at  every  level  is  closed  out  by  appro¬ 
priate  action;  i.e.,  redesign,  procurement, 
retest,  etc. 

•  Plan  Tests  of  Whole  System.  Plan  tests  of 
the  whole  system  including  proper  phas¬ 
ing  of  the  platform  and  supporting  gear, 
the  launcher,  the  missile  and  user  partici¬ 
pation. 

•  Determination  of  Component  Configura¬ 
tion.  Conditions  and  component  configu¬ 
ration  during  development  tests  should  be 
determined  by  the  primary  objectives  of 
that  test.  Whenever  a  nonoperational 
configuration  is  dictated  by  early  test  re- 
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quirements,  tests  should  not  be  challenged 
by  the  fact  that  configuration  is  not  opera¬ 
tional. 

•  Testing  of  Software.  Test  and  evaluation 
should  ensure  that  software  products  are 
tested  appropriately  during  each  phase. 
Software  often  has  been  developed  more  as 
an  add-on  than  as  an  integral  part  of  the 
overall  system.  Software  requirements  need 
the  same  consideration  as  hardware  re¬ 
quirements  in  the  PDRR  Phase. 

•  Range  Safety  Dry  Runs.  Ensure  the  test 
plan  includes  adequate  test  program/range 
safety  dry  runs.  The  government  test  ranges 
have  to  provide  facilities  to  safely  test  many 
different  projects. 

•  Assemblies/Subsystems  Special  Re¬ 
quirements, 

•  Seekers  and  tracking  devices, 

•  Propulsion  subsystems, 

•  Connectors  and  their  related  hard¬ 
ware, 

•  Lanyard  assemblies, 

•  Safeing,  arming,  fuzing  and  other 

ordnance  devices. 

•  Review  of  Air-to-Surface  Missile  (ASM) 
Test  Position  Fixes.  Review  the  final  posi¬ 
tion  fix  planned  before  launching  ASMs. 
There  are  instances  in  which  the  opera¬ 
tional  test  of  air-launched  missiles  uti¬ 
lized  artificial  position  fixes  just  prior  to 
missile  launch. 

•  Operator  Limitations.  Ensure  operator 
limitations  are  included  in  the  tests.  Most 
tactical  missiles,  especially  those  used  in 
close  support,  require  visual  acquisition  of 
the  target  by  the  missile  operator  and/or 
an  air/ground  controller. 


•  Test  Simulations  and  Dry  Runs.  Plan  and 
use  test  simulations  and  dry  runs.  Dry  runs 
should  be  conducted  for  each  new  phase  of 
testing.  Simulation  and  other  laboratory  or 
ground  testing  should  be  conducted  to  pre¬ 
dict  the  specific  test  outcome.  The  "wet 
run"  test  should  finally  be  run  to  verify  the 
test  objectives.  Evaluation  of  the  simula¬ 
tion  versus  the  actual  test  results  will  help 
to  refine  the  understanding  of  the  system. 

•  Component  Performance  Records.  Keep 
performance  records  on  components.  There 
are  many  examples  in  missile  programs 
that  have  required  parts  stock  sweeps  asso¬ 
ciated  with  flight  failures  and  component 
aging  test  programs. 

•  Tracking  Test  Data.  Ensure  the  test  pro¬ 
gram  tracks  data  in  a  readily  usable  man¬ 
ner.  Reliability  and  performance  evalua¬ 
tions  of  a  missile  system  should  break  down 
the  missile's  activity  into  at  least  the  follow¬ 
ing  phases: 

•  Prelaunch  including,  captive 

carry  reliability, 

•  Launch, 

•  In-flight, 

•  Accuracy /fuzing. 

•  Updating  lOT&E  Planning.  Periodi¬ 
cally  update  production  qualification  test¬ 
ing  MPE  and  lOT&E  planning  during  the 
early  R&D  phase.  Few  missile  system  po- 
grams  have  had  adequate  user  participa¬ 
tion  with  the  desired  continuity  of  per¬ 
sonnel  to  minimize  the  problems  of  tran¬ 
sition  from  DT&E  to  OT&E  to  deploy¬ 
ment/  utilization. 

•  Instrumentation  Provisions  in  Production 
Missiles.  Encourage  built-in  instrumenta¬ 
tion  provisions  in  production  missiles. 
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•  Constraints  on  Missile  Operator.  Detailed 
test  plans  should  be  evaluated  to  deter¬ 
mine  that  the  test  imposed  constraints  on 
the  missile  operator  do  not  invalidate  the 
applicability  of  the  data  so  collected. 

•  Problem  Fixes  Before  Production.  Ensure 
that  operational  suitability  tests  identify 
operational  deficiencies  of  new  systems 
quickly  so  fixes  can  be  developed  and  tested 
before  large-scale  production. 

•  Flight  Tests  Representative  of  Operations. 
Ascertain  that  final  DT&E  system  tests  and 
lOT&E  flight  tests  are  representative  of 
operational  flights.  Some  ballistic  missile 
R&E  programs  have  shown  high  success 
rates  in  R&E  flight  tests;  however,  when 
the  early  production  systems  were  de¬ 
ployed,  they  exhibited  a  number  of  unsat¬ 
isfactory  characteristics  such  as  poor  alert 
reliability  and  poor  operational  test-flight 
reliability. 

•  System  Interfaces  in  Operational  Test. 
Ensure  the  primary  objective  of  an  opera¬ 
tional  test  is  to  obtain  measurements  on  the 
overall  performance  of  the  weapon  system 
when  it  is  interfaced  with  those  systems 
required  to  operationally  use  the  weapons 
system. 

•  Realistic  Conditwnsfor  Operational  Testing. 
Ascertain  operational  testing  is  conducted 
under  realistic  combat  conditions.  This 
means  that  the  offense/defense  battle 
needs  to  be  simulated  in  some  way  before 
the  weapon  system  evaluation  can  be  con¬ 
sidered  completed.  Whether  this  exercise 
is  conducted  within  a  single  Service  (as  in 
the  test  of  a  surface-to-surface  antitank 
missile  against  tanks)  or  among  Services 
(as  in  the  test  of  an  air-to-surface  missile 
against  tanks  with  antiaircraft  protection), 
the  plans  for  such  testing  should  be  for¬ 
mulated  as  part  of  the  system  develop¬ 
ment  plan. 


25.2.2.4  Production ,  Fielding/ 
Deployment  and  Operational 
Support  Phase 

•  Testing  All  Operational  Modes.  Ensure 
the  FOT&E  plan  includes  tests  of  any  op¬ 
erational  modes  not  previously  tested  in 
lOT&E.  All  launch  modes  including  de¬ 
graded,  backup  modes  should  be  tested  in 
the  FOT&E  because  the  software  interface 
with  the  production  hardware  system 
should  be  evaluated  thoroughly.  Other¬ 
wise,  small,  easy-to-fix  problems  might 
preclude  launch. 

•  Extension  of  the  FOT&E  for  New  Threats. 
Be  alert  to  the  need  to  extend  the  FOT&E  if 
a  new  threat  arises.  Few  missile  programs 
perform  any  kind  of  testing  relatable  to 
evaluating  system  performance  against 
current  or  new  threats. 

•  "Lead-the-Fleet"  Production  Scheduling. 
Lead-the-Fleet  missile  scheduling  and  tests 
should  be  considered. 

•  Test  Fixes.  Test  fixes  result  from  earlier 
operational  testing.  After  the  lOT&E  that 
identified  problem  areas  in  missiles,  FOT &E 
should  evaluate  these  areas  primarily  to 
determine  the  adequacy  of  the  incorpo¬ 
rated  fixes,  particularly  if  the  lOT&E  did 
not  run  long  enough  to  test  the  fixes. 

•  FOT&E  Feedback  to  Acceptance  Testing. 
Ensure  that  FOT&E  results  are  quickly  fed 
back  to  influence  early  production  accep¬ 
tance  testing.  Production  acceptance  test¬ 
ing  is  probably  the  final  means  the  govern¬ 
ment  normally  will  have  to  ensure  the  prod¬ 
uct  meets  specifications.  Early  acceptance 
testing  could  be  influenced  favorably  by  a 
quick  feedback  from  FOT&E  to  acceptance 
testing.  This  is  exemplified  by  a  current 
ASM  program  where  production  has 
reached  peak  rates,  and  the  lOT&E  has  not 
been  completed. 
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25.2.3  Command  and  Control  Systems 
25.2.3.1  Concept  Exploration  Phase 

•  Concept  Test  Philosophy.  The  T&E  plan¬ 
ners  must  understand  the  nature  of  com¬ 
mand  and  control  (C^)  systems  early  in 
the  Concept  Exploration  Phase.  In  a  com¬ 
plex  command  and  control  system,  a  to¬ 
tal  systems  concept  must  be  developed 
initially.  Total  systems  life  cycle  must  be 
analyzed  so  the  necessary  requirement 
for  the  design  can  be  established. 

•  The  Importance  of  Software  Testing. 
Testers  should  recognize  that  software  is 
a  pacing  item  in  command  and  control 
systems  development. 

•  Software  Test  Scheduling  -  Contractors ' 
Facilities.  Provision  should  be  made  for 
including  software  T&E  during  each 
phase  of  C^  systems'  acquisition.  Avail¬ 
ability  of  contractors'  facilities  should  be 
considered. 

•  Evaluation  of  Exploratory  Development 
Tests.  Care  should  be  exercised  in  evaluat¬ 
ing  results  of  tests  conducted  during  ex¬ 
ploratory  development  of  command  and 
control  systems.  These  tests,  which  most 
likely  have  been  conducted  on  brassboard, 
breadboard  or  modified  existing  hardware, 
should  be  evaluated  with  special  attention. 

•  Feasibility  Testing  for  Field  Compilers. 
Early  test  planning  should  allow  for  simu¬ 
lating  the  computer  system  to  test  for 
field  use  of  compilers,  where  applicable. 

•  Evaluation  of  Test  Plan  Scheduling.  Mile¬ 
stones  should  be  event-oriented,  not  cal¬ 
endar-oriented. 

•  Type  Personnel  Needs  -  Effects  on  T&E. 
A  mix  of  personnel  with  different  back¬ 
grounds  affecting  T&E  is  required. 


•  Planning  for  Joint-Service  OT&E  Before 
Milestone  I.  Joint-Service  OT&E  (multi- 
Service)  should  be  considered  for  com¬ 
mand  and  control  systems. 

25.2.3.2  Program  Definition 
and  Risk  Reduction  Phase 

•  Test  Prototypes.  In  C^  systems,  proto¬ 
types  must  reasonably  resemble  final 
hardware  configuration  from  a  func¬ 
tional-use  standpoint.  When  high  techni¬ 
cal  risk  is  present,  development  should 
be  structured  around  the  use  of  one  or 
more  test  prototypes  designed  to  prove 
the  system  concept  under  realistic  opera¬ 
tional  conditions  before  proceeding  to 
engineering  development. 

•  Test  Objectives  —  Critical  Issues.  In 
addition  to  addressing  critical  technical 
issues,  T&E  objectives  during  the  PDRR 
Phase  should  address  the  functional  is¬ 
sues  of  a  C2  system. 

•  Real-Time  Software — Demonstration  of 
"Application  Patches."  Tests  of  real-time 

systems  should  include  demonstra¬ 
tions  of  interfaces  whereby  locally  gener¬ 
ated  application  patches  are  brought  into 
being. 

•  Independent  Software  Test-User  Group. 
An  independent  test-user  software  group 
is  needed  during  early  software  qualifi¬ 
cation  testing. 

•  System  Interfaces.  Critical  attention 
should  be  devoted  to  testing  interfaces 
with  other  systems  and  to  interfaces 
between  subsystems.  Particular  attention 
should  be  devoted  to  interfaces  with  other 

systems  and  to  the  interfaces  between 
sensors  (e.g.,  radar  units),  communica¬ 
tions  systems  (e.g.,  modems)  and  the  spe¬ 
cific  processors  (e.g.,  CPUs).  Interface 
with  information  processing  C^  systems 
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must  also  address  data-element  and  code¬ 
standardization  problems  if  data  is  to  be 
processed  on-line. 

•  Human  Factors.  Ina  O  system,  human 
factors  must  be  considered  from  the  ear¬ 
liest  prototype  designs  and  testing  pro¬ 
vided.  Testing  should  be  conducted  to 
determine  the  most  efficient  arrangement 
of  equipment  from  the  human  factor 
standpoint;  e.g.,  displays  should  be  ar¬ 
ranged  for  viewing  from  an  optimum 
angle  whenever  possible;  adequate  ma¬ 
neuvering  room  within  installation  con¬ 
straints  should  be  allowed  considering 
the  number  of  personnel  normally  man¬ 
ning  the  facility;  and  console-mounted 
controls  should  be  designed  and  located 
to  facilitate  operation,  minimize  fatigue 
and  avoid  confusion. 

•  Degraded  Operations  Testing.  When 
the  expected  operational  environment  of 
a  system  suggests  that  the  system  may 
be  operated  under  less-than-finely-tuned 
conditions,  tests  should  be  designed  to 
allow  for  performance  measurements 
under  degraded  conditions. 

•  Test-Bed.  The  use  of  a  test-bed  for  study 
and  experimentation  with  new  systems 
is  needed  early  in  the  PDRR  Phase. 

•  Software-Hardware  Interfaces.  The  soft¬ 
ware-hardware  interfaces,  with  all  opera¬ 
tional  backup  modes  to  a  new  system, 
should  be  tested  early  in  the  program. 

•  Reproducible  Tests.  Test  plans  should 
contain  a  method  for  allowing  full-load 
message  input  while  maintaining  repro¬ 
ducible  test  conditions. 

•  Cost-Effectiveness.  Field-test  data  is 
needed  during  the  PDRR  Phase  for  input 
to  cost-effectiveness  analyses  of  sys¬ 
tems. 


25.2.3.3  Engineering 
and  Manufacturing 
Development  Phase 

•  Acquisition  Strategi/.  The  acquisition 
strategy  for  the  system  should: 

•  Allow  sufficient  time  between  the 
planned  end  of  demonstration  testing 
and  major  procurement  (as  opposed 
to  limited  procurement)  decisions.  This 
provides  flexibility  for  modifying 
plans,  which  may  be  required  during 
the  test  phases  of  the  program.  For 
instance,  because  insufficient  time  was 
allowed  for  testing  one  recent  O  sys¬ 
tem,  the  program  and  the  contract  had 
to  be  modified  and  renegotiated; 

•  Be  evaluated  relative  to  constraints 
imposed; 

•  Ensure  that  sufficient  dollars  are 
available,  not  only  to  conduct  the 
planned  T &E  but  to  allow  for  the  addi¬ 
tional  T&E  that  is  always  required  due 
to  failures,  design  changes,  etc. 

•  Problem  Indications.  It  is  important 
to  establish  an  early  detection  scheme 
so  management  can  determine  when  a 
program  is  becoming  "ill." 

•  Impact  of  Software  Failures.  Prior  to 
any  production  release,  the  impact  of  soft¬ 
ware  failures  on  overall  system  perfor¬ 
mance  parameters  must  be  considered. 

•  Critical  Issues.  lOT&E  should  provide 
the  answers  to  some  critical  issues  pecu¬ 
liar  to  systems.  Some  critical  issues 
that  lOT&E  of  systems  should  answer 
are: 

•  Is  system  mission  reaction  time  a 
significant  improvement  over  present 
systems? 
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•  Is  a  backup  mode  provided  for  use 
when  either  airborne  or  ground  sys¬ 
tem  exhibits  a  failure? 

•  Can  the  system  be  transported  as 
operationally  required  by  organic 
transport?  (Consider  ground,  air  and 
amphibious  requirements.) 

•  Is  there  a  special  requirement  for  site 
preparation?  (For  example,  survey  and 
antenna  siteing.) 

•  Can  the  system  be  erected  and  dis¬ 
mantled  in  times  specified?  Are  these 
times  realistic? 

•  Does  relocation  affect  system  align¬ 
ment? 

•  Does  system  provide  for  operation 
during  maintenance? 

•  Can  on-site  maintenance  be  per¬ 
formed  onshelterless  subsystems  (e.g., 
radar  units)  during  adverse  weather 
conditions? 

•  Displays.  The  display  subsystems  of  a 
C^  system  should  provide  an  essential  func¬ 
tion  to  the  user.  Displays  are  key  subsystems 
of  a  C^  system.  They  provide  the  link  that 
couples  the  operator  to  the  rest  of  the  sys¬ 
tem  and  are,  therefore,  often  critical  to  its 
success. 

•  Pilot  Test.  A  pilot  test  should  be  con¬ 
ducted  before  lOT&E  so  sufficient  time  is 
available  for  necessary  changes. 

•  Publications  and  Manuals.  It  is  imperative 
that  all  system  publications  and  manuals  be 
completed,  reviewed  and  selectively  tested 
under  operational  conditions  before  begin¬ 
ning  overall  system  suitability  testing. 

•  Power  Sources.  Mobile,  prime  power 
sources  are  usually  provided  as  govern¬ 


ment-furnished  equipment  (GFE)  and  can 
be  a  problem  area  in  testing  C^  systems. 

•  lOT&E  Reliability  Data.  The  lOT&E  can 
provide  valuable  data  on  the  operational 
reliability  of  a  O  system;  this  data  cannot 
be  obtained  through  DT&E. 

•  Subsystem  Tests.  Every  major  subsystem 
of  a  C^  system  should  have  a  successful 
DT&E  before  beginning  overall  system 
operational  testing. 

•  Communications.  The  C^  systems  must 
be  tested  in  the  appropriate  electromag¬ 
netic  environment  to  determine  the  perfor¬ 
mance  of  its  communications  system. 

•  Maintenance.  In  lOT&E,  maintenance 
should  include:  a  measurement  of  the  ad¬ 
equacy  of  the  maintenance  levels  and  the 
maintenance  practices;  an  assessment  of 
the  impact  that  the  maintenance  plan  has 
on  the  operational  reliability;  the  accessi¬ 
bility  of  the  major  components  of  the  sys¬ 
tem  for  field  mainteriance  (e.g.,  cables  and 
connectors  are  installed  to  facilitate  access); 
and  verification  that  the  software  design 
for  maintenance  and  diagnostic  routines 
and  procedures  are  adequate  and  the  soft¬ 
ware  canbe  modified  to  accommodate  func¬ 
tional  changes. 

•  Continuity  of  Operations.  The  lOT&E 
should  provide  for  an  impact  assessment 
of  the  failure  of  any  subsystem  element  of 
a  O  system  on  overall  mission  effective¬ 
ness. 

•  Imitative  Deception.  The  lOT&E  should 
provide  for  tests  to  assess  the  susceptibility 
of  the  data  links  of  a  system  to  imitative 
deception. 

•  Demonstration  of  Procedures.  Test  plans 
should  include  a  procedural  demonstra¬ 
tion  whereby  the  tested  O  system  works  in 
conjunction  with  other  systems. 
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•  Government-Furnished  Equipment 
and  Facilities.  Test  and  evaluation  should 
be  concerned  about  the  availability  of 
GFE  equipment  as  specified  in  the  pro¬ 
posed  contract. 

•  User  Participation  in  T&E.  The  varying 
needs  of  the  user  for  a  C2  system  make 
participation  in  all  phases  of  T&E  manda¬ 
tory. 

25.2.3.4  Production,  Fielding/ 
Deployment  and  Operational 
Support  Phase 

•  First  Article  Testing.  The  preproduction, 
first  article  testing  and  evaluation  should 
be  designed  and  conducted  to:  (1)  confirm 
the  adequacy  of  the  equipment  to  meet 
specified  performance  requirements;  (2) 
confirm  the  adequacy  of  the  software  not 
only  to  meet  current  user  needs  but  to 
accommodate  changing  needs;  and  (3)  de¬ 
termine  failure  modes  and  rates  of  the  total 
integrated  system.  This  activity  should  be 
followed  by  FOT&E. 

•  Test  Planners  and  Evaluators.  Use  the 
lOT&E  personnel  in  the  FOT&E  program. 
The  planners  and  evaluators  for  the  FOT&E 
of  the  production  system  can  do  a  better  job 
if  they  are  involved  initially  in  planning 
and  conducting  the  lOT&E. 

25.2.4  Ship  Systems 

25.2.4.1  Concept 
Exploration  Phase 

•  Test  and  Evaluation  Master  Plan.  Prior  to 
Milestone  I,  sufficient  materiel  should  be 
generated  to  allow  for  evaluating  the  over¬ 
all  T&E  program. 

•Test  Objectives  and  Critical  Issues.  In 
evaluating  the  initial  test  concept,  it  is  im¬ 
portant  that  the  test  objectives  during  the 
time  from  Milestone  I  to  Milestone  II  ad¬ 


dress  the  major  critical  issues,  especially 
technological  issues. 

•  Test  Facilities  and  Instrumentation  Re¬ 
quired.  Before  Milestone  I,  the  test  facilities 
and  instrumentation  requirements  to  con¬ 
duct  developmental  and  operational  tests 
and  a  tentative  schedule  of  test  activities 
should  be  identified. 

•  Multiple  Approach  To  Weapon  System 
Development.  Whenever  possible,  the 
weapon  system  concept  should  not  be 
predicated  on  the  successful  development 
of  a  single  hardware  or  software  approach 
in  the  various  critical  subsystems  (unless  it 
has  been  previously  demonstrated  ad¬ 
equately). 

•  Comparison  of  New  versus  Old  System. 
The  procedure  for  examining  the  relative 
performance  of  new  or  modified  systems 
versus  old  should  be  indicated  in  the  T &E 
plan. 

•  Test  Support  Facilities.  The  phasing  of 
test  support  facilities  must  be  planned  care¬ 
fully,  with  some  schedule  flexibility  to  cover 
late  delivery  and  other  unforeseen  prob¬ 
lems. 

•  Fleet  Operating  Force  Requirements.  The 
requirement  for  fleet  operating  forces  for 
DT &E  or  OT &E  should  be  assessed  early  in 
the  program  and  a  specific  commitment 
made  as  to  the  types  of  units  to  be  em¬ 
ployed. 

•  Mission-Related  Measures  of  Effective¬ 
ness.  During  the  Concept  Exploration  Phase 
of  the  acquisition  of  a  new  class  of  ship,  a 
study  effort  should  be  commenced  jointly 
by  the  Chief  of  Naval  Operations  (CNO) 
and  the  Commander,  Operational  Test  and 
Evaluation  Force  (COMOPTEVFOR).  This 
effort  is  to  establish  mission-related  mea¬ 
sures  of  effectiveness,  which  may  be  ex¬ 
pressed  in  numerical  fashion  and  may 
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later  be  made  the  subject  of  OT&E  to 
determine  how  closely  the  new  ship  sys¬ 
tem  meets  the  operational  need  for  which 
it  was  conceived. 

•  Ship  T&E  Management.  The  manage¬ 
ment  of  ship  T&E  should  ensure  that  test 
requirements  are  necessary  and  consistent 
relative  to  systems/ subsystem  aspects  and 
that  the  necessary  testing  is  coordinated  so 
that  test  redundancy  does  not  become  a 
problem. 

•  T&E  of  Large,  Integrally-Constructed  Sys¬ 
tems.  Major  subsystems  should  be  proven 
feasible  before  firm  commitment  to  a  de¬ 
tailed  hull  design. 

25.2.4.2  Program  Definition 
and  Risk  Reduction  Phase 

•  Authentication  of  Human  Factors  Con¬ 
cepts.  Test  and  evaluation  should  authenti¬ 
cate  the  human  factors  concepts  embodied 
in  the  proposed  systems  design,  examining 
questions  of  safety,  comfort,  appropriate¬ 
ness  of  man-machine  interfaces,  as  well  as 
the  numbers  and  skill  levels  of  the  person¬ 
nel  required. 

•  Acquisition  Strategy.  The  acquisition 
strategy  for  a  ship  and  its  subsystems  should 
allow  sufficient  time  between  the  plarmed 
end  of  demonstration  testing  and  major 
procurement  decisions  of  government-fur¬ 
nished  equipment  for  flexibility  to  modify 
plans  (may  be  required  during  the  test 
phases  of  the  program). 

•  Evaluation  of  Results  of  Exploratory  Test¬ 
ing.  Results  of  tests  conducted  during  ex¬ 
ploratory  development  and  most  likely 
conducted  onbrassboards,  breadboards  or 
modified  existing  hardware  should  be 
evaluated  carefully. 

•  Software  Testing.  In  view  of  increased 
dependence  upon  computers  in  ship  man¬ 


agement  and  tactical  operation,  software 
testing  must  be  exceptionally  thorough, 
and  integrated  software  testing  must  begin 
as  early  as  possible. 

•  New  Hull  Forms.  When  a  new  t5q?e  of 
ship  involves  a  radical  departure  from  the 
conventional  hull  form,  extensive  proto- 
t5q?e  testing  should  be  required  prior  to 
further  commitment  to  the  new  hull  form. 

•  Effects  of  Hull  and  Propulsion  on  Mission 
Capability.  The  predicted  effects  of  the 
proven  hull  and  propulsion  system  design 
on  the  performance  of  the  ship's  critical 
system  should  be  determined. 

•  Advances  in  Propulsion.  Demonstration 
of  the  use  of  new  propulsion  systems  should 
be  conducted  prior  to  making  the  decision 
to  commit  the  propulsion  systems  to  the 
ship  in  question. 

•  Propulsion  Systems  in  Other  Classes. 
When  an  engine  to  be  used  in  the  propul¬ 
sion  system  of  a  new  ship  is  already  per¬ 
forming  satisfactorily  in  another  ship,  this 
is  not  to  be  taken  as  an  indication  that 
shortcuts  can  be  taken  in  propulsion  sys¬ 
tem  DT&E,  or  that  no  problems  will  be 
encountered. 

•  The  OT&E  of  Shipboard  Gun  Systems. 
Operational  tests  of  shipboard  gun  sys¬ 
tems  should  simulate  the  stress,  expo¬ 
sure  time  and  other  conditions  of  battle 
so  that  the  suitability  of  the  weapon  can 
be  evaluated  in  total. 

•  Targets  for  Antiaircraft  Warfare  (AAW) 
OT&E.  Operational  test  of  shipboard 
AAW  weapons  demands  the  use  of  tar¬ 
gets  which  realistically  simulate  the 
present-day  threat. 

•  Waivers  to  T&E  of  Ship  Systems.  Waiv¬ 
ers  to  T&E  of  preproduction  models  of  a 
system  in  order  to  speed  up  production 
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and  delivery  should  be  made  only  after 
considering  all  costs  and  benefits  of  the 
waiver,  including  those  not  associated 
with  the  contract. 

•  Environment  Effects  on  Sonar  Domes, 
Environmental  effects  on  sonar  domes 
and  their  self-noise  should  be  tested  and 
evaluated  before  the  domes  are  accepted 
as  part  of  the  sonar  system. 

•  Hull /Machinery  Testing  by  Computer 
Simulation.  In  DT&E  ships,  there  will  be 
cases  where  the  best  means  to  conduct 
evaluations  of  particular  hull  and 
machinery  capabilities  is  through 
dynamic  analysis  using  computer 
simulation,  with  later  validation  of  the 
simulation  by  actual  test. 

•  Operational  Reliability.  The  OT&E 
should  provide  valuable  data  on  the  op¬ 
erational  reliability  of  ship  weapon  sys¬ 
tems  that  cannot  be  obtained  through 
DT&E. 

25.2.4.3  Engineering 
and  Manufacturing 
Development  Phase 

•  Initial  or  Pilot  Phase  oflOT&E.  Before 
any  operational  tests  to  demonstrate  op¬ 
erational  suitability  and  effectiveness  are 
conducted,  an  initial  or  pilot  test  should 
be  conducted. 

•  Identify  Critical  Subsystems.  In  plan¬ 
ning  for  the  lOT&E  of  a  ship  system,  the 
critical  subsystems,  with  respect  to  mis¬ 
sion  performance,  should  be  identified. 

•  Reliability  of  Critical  Systems.  Test  and 
evaluation  should  determine  the  expected 
reliability  at  sea  of  systems  critical  to  the 
ship's  mobility  and  to  the  primary  and 
major  secondary  tasks. 


•  Consistency  in  Test  Objectives.  There 
are  various  phases  in  testing  a  ship  sys¬ 
tem.  One  should  ensure  the  objectives  of 
one  phase  are  not  inconsistent  with  the 
objectives  of  the  other  phases. 

•  Single  Screw  Ships.  Test  and  evalua¬ 
tion  of  the  propulsion  systems  of  ships 
with  a  single  screw  should  be  especially 
rigorous  to  determine  failure  rates,  main¬ 
tenance  and  repair  alternatives. 

•  Problems  Associated  With  New  Hulls. 
Whenever  a  new  hull  is  incorporated  into 
ship  design,  a  T&E  of  this  hull  should  be 
conducted  prior  to  the  full-rate  produc¬ 
tion  and  incorporation  of  the  major  weap¬ 
ons  subsystems. 

25.2.4.4  Production,  Fielding/ 
Deployment  and  Operational 
Support  Phase 

•  Design  of  Ship  FOT&E.  In  the  testing 
program  of  a  ship  system,  it  should  be 
recognized  that,  although  it  may  be  des¬ 
ignated  as  a  special-purpose  ship,  in  most 
cases  it  will  be  used  in  a  general-purpose 
role  as  well. 

•  Operational  Testing  During  Shakedown 
Periods.  The  time  period  for  FOT&E  of  a 
ship  can  be  used  more  efficiently  if  full 
advantage  is  taken  of  the  periods  immedi¬ 
ately  after  the  ship  is  delivered  to  the  Navy. 

•  Fleet  Operations  in  FOT&E.  A  great 
deal  of  information  on  the  operational 
effectiveness  of  a  ship  can  be  obtained 
from  standard  fleet  operations  through 
well-designed  information  collection, 
processing  and  analysis  procedures. 

•  Ship  Antisubmarine  Warfare  (ASW) 
FOT&E  Planning.  In  planning  FOT&E  of 
shipboard  systems,  it  is  important  to  recog¬ 
nize  the  difficulty  of  achieving  realism. 
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perhaps  more  so  than  in  other  areas  of 
naval  warfare. 

•  Variable  Depth  Sonar  FOT&E.  The  behav¬ 
ior  of  towed  bodies  of  variable  depth  sonar 
systems  and  towed  arrays  should  be  tested 
and  evaluated  under  all  ship  maneuvers  and 
speeds  likely  to  be  encountered  in  combat. 

•  Ship  Self-Noise  Tests.  The  magnetic  and 
acoustic  signatures  of  a  ship  can  be  tested 
accurately  only  after  it  is  completed. 

•  Effect  of  Major  Electronic  Countermeasures 
(ECMs)  on  Ship  Capability.  The  FOT&E  of  a 
ship  should  include  tests  of  the  effectiveness 
of  ^e  ship  when  subjected  to  major  ECM. 

•  Ship  System  Survivability.  Follow-on  Op¬ 
erational  Test  and  Evaluation  of  modern 
ships  should  provide  for  the  assessment  of 
their  ability  to  survive  and  continue  to  fight 
when  subjected  to  battle  damage. 

•  Interlocks.  Shipboard  electronic  systems 
are  designed  with  interlock  switches  that 
open  electrical  circuits  for  safety  reasons 
when  the  equipment  cabinets  are  opened. 
The  FOT&E  should  be  able  to  detect  over- 
design  as  well  as  minimum  design  ad¬ 
equacy  of  the  interlock  systems. 

•  Intraship  Communication.  In  conducting 
lead  ship  trials  and  evaluations,  particular 
attention  should  be  given  to  the  opera¬ 
tional  impact  resulting  from  absence,  by 
design,  of  intraship  communications  cir¬ 
cuits  and  stations  from  important  operat¬ 
ing  locations. 

25.2.5  Surface  Vehicle  Systems 
25.2.5.1  Concept  Exploration  Phase 

•  Preparing  Test  Plans.  It  is  necessary  that 
a  detailed  evaluation  criteria  be  established 
that  includes  all  items  to  be  tested. 


•  Test  Plans.  Prior  to  Milestone  I,  a  plan 
should  be  prepared  for  evaluating  the  over¬ 
all  T&E  program.  As  part  of  this,  a  detailed 
T&E  plan  for  those  tests  to  be  conducted 
before  Milestone  II  to  validate  the  concept 
and  hardware  approach  to  the  vehicle  sys¬ 
tem  should  be  developed.  The  objective  of 
the  validation  test  plan  is  to  fully  evaluate 
the  performance  characteristics  of  the  new 
concept  vehicle.  This  test  plan  cannot  be 
developed,  of  course,  until  the  performance 
characteristics  are  defined. 

•  Performance  Characteristics  Range. 
Stated  performance  characteristics  de¬ 
rived  from  studies  should  be  measured 
early  in  the  program.  Unrealistic  perfor¬ 
mance  requirements  can  lead  to  false 
starts  and  costly  delays. 

•  Operating  Degradation.  System  perfor¬ 
mance  degrades  under  field  conditions. 
Anticipated  degradation  must  be  consid¬ 
ered  during  T&E.  When  a  system  must 
operate  at  peak  performance  during  de¬ 
velopment  test/operational  test  (DT/OT) 
to  meet  the  specified  requirements,  it  will 
then  be  likely  to  perform  at  a  lesser  level 
when  operated  in  the  field. 

•  Test  Personnel.  The  test  director  and/ 
or  key  members  of  the  test  planning  group 
within  the  project  office  should  have  sig¬ 
nificant  T&E  experience. 

•  Design  Reviews.  T&E  factors  and  expe¬ 
rience  must  influence  the  system  design. 
The  application  of  knowledge  derived  from 
past  experience  can  be  a  major  asset  in 
arriving  at  a  sound  system  design. 

•  Surrogate  Vehicles.  When  high  techni¬ 
cal  risk  is  present,  development  should 
be  structured  around  the  use  of  one  or 
more  surrogate  vehicles  designed  to  prove 
the  system  concept  under  realistic  opera¬ 
tional  conditions  before  proceeding  with 
further  development. 
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•  Test  Facilities  and  Scheduling.  Before 
Milestone  I,  test  range  and  resource  re¬ 
quirements  to  conduct  validation  tests  and 
a  tentative  schedule  of  test  activities  should 
be  identified. 

25.2.5.2  Program  Definition  and 
Risk  Reduction  Phase 

•  Vulnerability.  The  vulnerability  of  ve¬ 
hicles  should  be  estimated  on  the  basis  of 
testing. 

•  Gun  and  Ammunition  Performance.  Gun 
and  ammunition  development  should  be 
considered  a  part  of  overall  tank  system 
development.  When  a  new  gun  tube,  or  one 
which  has  not  been  mounted  previously  on 
a  tank  chassis,  is  being  evaluated,  all  ammu¬ 
nition  types  (including  missiles)  planned 
for  use  in  that  system  should  be  test  fired 
under  simulated  operational  conditions. 

•  Increased  Complexity.  The  addition  of 
new  capabilities  to  an  existing  system  or 
system  type  will  generally  increase  com¬ 
plexity  of  the  system  and,  therefore,  in¬ 
crease  the  types  and  amount  of  testing  re¬ 
quired  and  the  time  to  perform  these  tests. 

•  Component  Interfaces.  Prior  to  assembly 
in  a  protot)q)e  system,  component  sub¬ 
systems  should  be  assembled  in  a  mock-up 
and  verified  for  physical  fit,  human  factors 
considerations,  interface  compatibility  and 
for  electrical  and  mechanical  compatibil¬ 
ity. 

•  Determining  Test  Conditions.  During 
validation,  test  conditions  should  be  deter¬ 
mined  by  the  primary  objectives  of  that  test 
rather  thanby  more  general  considerations 
of  realism. 

•  Test  Plan  Development.  The  test  plan 
developed  by  this  point  should  be  in  nearly 
final  form  and  include,  as  a  minimum: 


•  A  description  of  requirements, 

•  The  facilities  needed  to  make  evalu¬ 
ations, 

•  The  schedule  of  evaluations  and  fa¬ 
cilities, 

•  The  reporting  procedure,  the  objec¬ 
tive  being  to  communicate  test  results 
in  an  understandable  format  to  all  pro¬ 
gram  echelons, 

•  The  T&E  guidelines,  and 

•  A  further  refinement  of  the  cost  esti¬ 
mates  which  were  initiated  during  the 
Concept  Evaluation  Phase. 

•  Prototype  Tests.  Prototype  tests  should 
show  satisfactory  meeting  of  success  crite¬ 
ria  which  are  meaningful  in  terms  of  opera¬ 
tional  usage.  It  is  essential  in  designing 
contractually  required  tests,  upon  whose 
outcome  large  incentive  payments  or  even 
program  continuation  may  depend,  to 
specify  broader  success  criteria  than  sim¬ 
ply  hit  or  miss  in  a  single  given  scenario. 

•  Reliability  Testing.  Reliability  testing 
should  be  performed  on  component  and 
subsystem  assemblies  before  testing  of  the 
complete  vehicle  system.  Prior  to  full  sys¬ 
tem  testing,  viable  component  and  sub¬ 
system  tests  should  be  conducted. 

•  Human  Factors.  In  evaluating  ground 
vehicles,  human  factors  should  be  consid¬ 
ered  at  all  stages  starting  with  the  design  of 
the  prototype. 

•  Test  Plan  Scheduling.  Test  plan  schedul¬ 
ing  should  be  tied  to  event  milestones  rather 
than  to  the  calendar.  In  evaluating  the  ad¬ 
equacy  of  the  scheduling  as  given  by  test 
plans,  it  is  important  that  milestones  be  tied 
to  the  major  events  of  the  weapon  system 
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(meeting  stated  requirements)  and  not 
the  calendar. 

•  Test  Failures.  The  T&E  schedule  should 
be  sufficiently  flexible  to  accommodate 
failures  and  correction  of  identified  prob¬ 
lems. 

25.2.5.3  Engineering  and 
Manufacturing  Development  Phase 

•  Planning  the  lOT&E.  The  lOT&E 
should  be  cost-effective  and  provide 
meaningful  results. 

•  Pilot  and  Dry-Run  Tests.  A  scheduled 
series  of  tests  should  be  preceded  by  a 
dry  run,  which  verifies  that  the  desired 
data  will  be  obtained. 

•  Comparison  Testing.  The  test  program 
should  include  a  detailed  comparison  of 
the  characteristics  of  a  new  vehicle  system 
with  those  of  existing  systems,  alternate 
vehicle  system  concepts  (if  applicable)  and 
those  of  any  system(s)  being  replaced. 

•  Simulation.  Simulation  techniques 
and  equipment  should  be  utilized  to 
enhance  data  collection.  Creation  of  his¬ 
tograms  for  each  test  course  provides  a 
record  of  conditions  experienced  by  the 
vehicle  during  testing.  Use  of  a  chassis 
dynamometer  can  produce  additional 
driveling  endurance  testing  with  more 
complete  instrumentation  coverage. 

•  Environmental  Testing.  Ground  ve¬ 
hicles  should  be  tested  in  environmental 
conditions  and  situations  comparable  to 
those  in  which  they  will  be  expected  to 
perform. 

•  System  Vulnerability.  For  combat  ve¬ 
hicles,  some  estimate  of  vulnerability  to 
battle  damage  should  be  made. 


•  Design  Criteria  Verification.  Subsystem 
design  criteria  should  be  compared  with 
actual  characteristics. 

•  Electromagnetic  Testing.  Vehicle  testing 
should  include  electromagnetic  testing. 

•  System  Strength  Testing.  In  evaluating 
ground  vehicles,  early  testing  should 
verify  intrinsic  strength.  This  implies 
operation  with  maximum  anticipated 
loading,  including  trailed  loads  at  maxi¬ 
mum  speeds  and  over  worst-case  grades, 
secondary  roads  and  cross-country  con¬ 
ditions  for  which  the  vehicle  was  devel¬ 
oped  or  procured.  This  test  is  intended  to 
identify  deficient  areas  of  design,  not  to 
break  the  machinery. 

•  Component  Compatibility.  Component 
compatibility  should  be  checked  through 
the  duration  of  the  test  sequence. 

•  Human  Interface.  Critiques  of  good 
and  bad  features  of  the  vehicle  should  be 
made  early  in  the  prototype  stage  while 
adequate  time  remains  to  make  any  indi¬ 
cated  changes. 

•  Serviceability  Testing.  Ground  vehicles 
should  be  tested  and  evaluated  to  deter¬ 
mine  the  relative  ease  of  serviceability, 
particularly  with  high-frequency  opera¬ 
tions. 

•  Experienced  User  Critique.  Ground  ve¬ 
hicle  user  opinions  should  be  obtained 
early  in  the  development  program. 

•  Troubleshooting  During  Tests.  Provi¬ 
sions  should  be  made  to  identify  subsystem 
failure  causes.  Subsystems  may  exhibit 
failures  during  testing.  Adequate  provi¬ 
sions  should  be  made  to  permit  trouble¬ 
shooting  and  identification  of  defective 
components  and  inadequate  design. 
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25.2.5.4  Production,  Fielding/ 
Deployment  and  Operational 
Support  Phase 


model  vehicle  should  be  allocated  to  inten¬ 
sive  testing  to  accumulate  high  operating 
time  in  a  short  period. 


•  Performance  and  Reliability  Testing.  The 
production  first-article  testing  should  verify 
the  performance  of  the  vehicle  system  and 
determine  the  degradation,  failure  modes 
and  failure  rates. 

•  Lead-the-Fleet  Testing.  At  least  one  pro¬ 
duction  prototype  or  initial  production 


•  User  Evaluation.  User-reported  short¬ 
comings  should  be  followed  up  to  deter¬ 
mine  problem  areas  requiring  correction. 
Fixes  should  be  evaluated  during  an 
FOT&E. 
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APPENDIX  A 

ACRONYMS  AND  THEIR  MEANINGS 


AAA 

AAE 

AAH 

ACAT 

ACM 

ADA 

ADATS 

ADM 

AFEWES 

AFMC 

AFOTEC 

AF/TE 

ALCM 

AMC 

AMARC 

AMSDL 

AOA 

APB 

ASAF(A) 
ASA  (RD&A) 
ASD(PAE) 
ASN  (RD&A) 
ASN/RE&S 

AID 

ATE 

AWACS 

BIS 

BIT 

BITE 

BLRIP 

BoD 

BoOD 

O 

ai 

ai 

OISR 

CDR 

CDRL 


Army  Audit  Agency 
Army  Acquisition  Executive 
Advanced  Attack  Helicopter 
Acquisition  Category 
Advanced  Cruise  Missile 
Acquisition  Decision  Authority 

Army  Development  and  Acquisition  of  Threat  Simulators 
Acquisition  Decision  Memorandum 
Air  Force  Electronic  Warfare  Evaluation  Simulator 
Air  Force  Materiel  Command 
Air  Force  Operational  Test  and  Evaluation  Center 
Air  Force/Test  and  Evaluation  Office 
Air  Launch  Cruise  Missile 
Army  Materiel  Command 
Army  Materiel  Acquisition  Review  Committee 
Acquisition  Management  Systems  and  Data  Requirements 
Control  List 

Analysis  of  Alternatives 

Acquisition  Program  Baseline 

Asst.  Secretary  of  Air  Force  (Acquisition) 

Asst.  Sec.  of  Army  (Research,  Dev.  and  Acquisition) 

Asst.  Sec.  of  Def.  (Program  Analysis  and  Evaluation) 

Asst.  Sec.  of  Navy  (Research,  Dev.  and  Acquisition) 

Assistant  Secretary  of  the  Navy/Research,  Engineering  and 
Science 

Advanced  Technology  Demonstration 
Automatic  Test  Equipment 
Airborne  Warning  and  Control  System 
Board  of  Inspection  and  Survey 
Built-in  Test 
Built-in  Test  Equipment 
Beyond  Low  Rate  Initial  Production  Report 
Board  of  Directors 
Board  of  Operating  Directors 
Command  and  Control 
Command,  Control  and  Communication 
Command,  Control,  Communications,  Intelligence 
Command,  Control,  Communications,  Computers  and  Intelligence 
Command,  Control,  Communications,  Intelligence,  Surveillance,  & 
Reconnaissance 
Critical  Design  Review 
Contract  Data  Requirements  List 
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CDS 

CE 

CED 

CEP 

CG  MCSC 
Cl 

CINC 

CLIN 

CNO 

CNP 

COCI 

COEA 

COI 

COMOPTEVFOR 

CSC 

CSCI 

CSTA 

CSU 

CTEIP 

DA 

DAB 

DAE 

DAG 

DBDD 

DCMC 

DCP 

DCSOPS 

DCS/R&D 

DDDR&E 

DEM/VAL 

DID 

DLT 

DMSO 

DNA 

DoD 

DoDD 

DoDI 

DOE 

DOT&E 

DPESO 

DPML 

DPRB 

DPRO 

DRB 

DSARC 

DSB 


Congressional  Data  Sheets 
Concept  Exploration  Phase 
Concept  Exploration/Definition  Phase 
Circle  Error  Probability 

Commanding  General,  Marine  Corps  Systems  Command 

Configuration  Item 

Fleet  Commander  in  Chief 

Contract  Line  Item  Number 

Chief  of  Naval  Operations 

Candidate  Nomination  Proposal 

Critical  Operational  Issues  and  Criteria 

Cost  and  Operational  Effectiveness  Analysis 

Critical  Operational  Issue 

Commander,  Operational  Test  and  Evaluation  Force  (Navy) 

Computer  Software  Component 

Computer  Software  Configuration  Item 

Combined  Systems  Test  Activity 

Computer  Software  Unit 

Central  Test  and  Evaluation  Investment  Program 
Developing  Agency  (Navy) 

Defense  Acquisition  Board 

Defense  Acquisition  Executive 

Data  Automation  Group 

Data  Base  Design  Document 

Defense  Contract  Management  Command 

Decision  Coordination  Paper 

Deputy  Chief  of  Staff  for  Operations  and  Plans 

Deputy  Chief  of  Staff  for  Research  and  Development 

Deputy  Director  Defense  Research  and  Engineering 

Demonstration/Validation  Phase 

Data  Item  Description 

Design  Limit  Test 

Defense  Modeling  and  Simulation  Office 
Defense  Nuclear  Agency 
Department  of  Defense 
Department  of  Defense  Directive 
Department  of  Defense  Instruction 
Department  of  Energy 
Director,  Operational  Test  and  Evaluation 
DoD  Product  Engineering  Services  Office 
Deputy  Program  Manager,  Logistics 
Defense  Planning  and  Resources  Board 
Defense  Plant  Representative  Office 
Defense  Resources  Board 

Defense  Systems  Acquisition  Review  Council  (now  DAB) 
Defense  Science  Board 
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DT 

Development  Test 

DT&E 

Development  Test  and  Evaluation 

DTSEE 

Director,  Test,  Systems  Engineering  and  Evaluation 

DTESG 

Defense  Test  and  Evaluation  Steering  Group 

DTTSG 

Defense  Test  and  Training  Steering  Group 

DUSA(OR) 

Deputy  Undersecretary  of  Army  (Gperations  Research) 

DVAL 

Data  Link  Vulnerability  Analysis 

EA 

Evolutionary  Acquisition 

EC 

Electronic  Combat 

ECCM 

Electronic  Counter-Countermeasures 

ECM 

Electronic  Countermeasures 

ECP 

Engineering  Change  Proposal 

ECR 

Engineering  Change  Review 

EDM 

Engineering  Development  Model 

EDT 

Engineering  Design  Test 

EMD 

Engineering  and  Manufacturing  Phase 

EMI 

Electromagnetic  Interference 

EMP 

Electromagnetic  Pulse 

EGA 

Early  Gperational  Assessment 

EGA,  GA 

Early  Gperational  Assessment,  Gperational  Assessment 

ERAM 

Extended  Range  Anti-armor  Munitions 

ESM 

Electronic  Support  Measures 

ESS 

Environmental  Stress  Screening 

EW 

Electronic  Warfare 

FAADS 

Forward  Area  Air  Defense  System 

FAR 

Federal  Acquisition  Regulation 

FAT 

First  Article  Testing 

FCA 

Functional  Configuration  Audit 

FDT&E 

Force  Development  Tests  and  Experimentation 

FFBD 

Functional  Flow  Block  Diagram 

FGRSCGM 

Forces  Command  (Army) 

FGT&E 

Follow-on  Gperational  Test  and  Evaluation 

FQR 

Formal  Qualification  Review 

FSD 

Full  Scale  Development  (now  EMD) 

FWE 

Foreign  Weapons  Evaluation 

FYDP 

Future  Years  Defense  Program 

FYTP 

FutureYears  Test  Program 

GPMG 

Government  Program  Management  Gffice 

HWCI 

Hardware  Configuration  Item 

HWIL 

Hardware-in- the-Loop 

ICBM 

Intercontinental  Ballistic  Missile 

IDD 

Interface  Decision  Document 

lEP 

Independent  Evaluation  Plan 

lER 

Independent  Evaluation  Report 

IFFN 

Identification,  Friend,  Foe,  Neutral 

IFPP 

Information  for  Proposal  Preparation 
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ILS 

ILSMT 

ILSP 

IOC 

lOT&E 

IPS 

IPX 

IRA 

IRS 

ITEA 

IIP 

IV&V 

JCG(T&E) 

JLF 

JORD 

JPO 

JRD 

JROC 

JT&E 

JTC3A 

LET 

LFT&E 

LRIP 

LSA 

LSAR 

MAA 

MAIS 

MAJCOM 

MCCR 

MCOTEA 

MDA 

MDAP 

MIL-SPEC 

MIL-STD 

MMOU 

MNS 

MOA 

MOE 

MOP 

MOU 

MPE 

MRTFB 

MS 

MSTIRC 

NAPMA 

NAVAIR 


Integrated  Logistics  Support 

Integrate  Logistics  Support  Management  Team 

Integrated  Logistics  Support  Plan 

Initial  Operating  Capability 

Initial  Operational  Test  and  Evaluation 

Integrated  Program  Summary 

Integrated  Product  Team 

Industrial  Resource  Analysis 

Interface  Requirements  Specification 

International  Test  and  Evaluation  Association 

Integrated  Test  Plan 

Independent  Verification  and  Validation 
Joint  Commanders  Croup  (T&E) 

Joint  Live  Fire 

Joint  Operational  Requirements  Document 

Joint  Program  Office 

Joint  Requirements  Document 

Joint  Requirements  Oversight  Council 

Joint  Test  and  Evaluation 

Joint  Tactical  Agency 

Live  Fire  Test 

Live  Fire  Test  and  Evaluation 
Low  Rate  Initial  Production 
Logistics  Support  Analysis 
Logistics  Support  Analysis  Report 
Mission  Area  Analysis 
Major  Automated  Information  System 
Major  Commands 

Mission  Critical  Computer  Resources 

Marine  Corps  Operational  Test  and  Evaluation  Agency 

Milestone  Decision  Authority 

Major  Defense  Acquisition  Program 

Military  Specification 

Military  Standard 

Multinational  Memorandum  of  Understanding 

Mission  Needs  Statement 

Memorandum  of  Agreement 

Measure  of  Effectiveness 

Measure  of  Performance 

Memorandum  of  Understanding 

Military  Preliminary  Evaluation 

Major  Range  and  Test  Facility  Base 

Milestone 

Multi-Service  Test  Investment  Resource  Committee 
North  Atlantic  Program  Management  Agency 
Naval  Air  Systems  Command 
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NAVSEA 

Naval  Sea  Systems  Command 

NAWC 

Naval  Air  Warfare  Center 

NBC 

Nuclear,  Biological,  and  Chemical 

NBCC 

Nuclear,  Biological,  and  Chemical  Contamination 

NDCP 

Navy  Decision  Coordinating  Paper 

NDI 

Nondevelopment  Item 

NEPA 

National  Environmental  Policy  Act 

NH&S 

Nuclear  Hardness  and  Survivability 

O&M 

Operations  and  Maintenance 

O&S 

Operations  and  Support 

OA 

Operational  Assessment 

OEC 

Operational  Evaluation  Command 

OIPT 

Overarching  Integrated  Product  Team 

OJOS 

Office  of  the  Joint  Chiefs  of  Staff 

OMB 

Office  of  Management  and  Budget 

OPEVAL 

Operational  Evaluation 

OPNAV 

Operational  Navy 

OPNAVIST 

Operational  Navy  Instruction 

OPSEC 

Operations  Security 

OPTEC 

Operational  Test  and  Evaluation  Command 

OPTEVFOR 

Operational  Test  and  Evaluation  Force 

ORD 

Operational  Requirement  Document 

ORMAS/TE 

Operational  Resource  Mgmt  Assessment  Systems  for  T&E 

OSD 

Office  of  the  Secretary  of  Defense 

OT&E 

Operational  Test  and  Evaluation 

OT 

Operational  Test 

OTA 

Operational  Test  Agency 

OTD 

Operational  Test  Director 

OTEA 

Operational  Test  and  Evaluation  Agency 

OTO 

Operational  Test  Organization 

OTP 

Outline  Test  Plan 

p3I 

Preplanned  Product  Improvements 

PAT&E 

Production  Acceptance  Test  and  Evaluation 

PCA 

Physical  Configuration  Audit 

PCO 

Primary  Contracting  Officer 

PDRR 

Program  Definition  and  Risk  Reduction 

PDR 

Preliminary  Design  Review 

PDSS 

Post-Deployment  Software  Support 

PEP 

Producibility  Engineering  Plan 

PI 

Product  Improvement 

PM 

Program  Manager 

PMO 

Program  Management  Office 

PO 

Program  Office,  Purchase  Order 

POM 

Program  Objectives  Memorandum 

PPBS 

Planning,  Programming,  and  Budgeting  System 

PPQT 

Preproduction  Qualification  Tests 
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PQT 

Production  Qualification  Test 

PRAT 

Production  Reliability  Acceptance  Test 

PRESINSURV 

President  of  the  Boards  of  Inspection  and  Survey 

PRR 

Production  Readiness  Review 

QOT&E 

Qualification  Operational  Test  and  Evaluation 

R&D 

Research  and  Development 

R&E 

Research  and  Engineering 

RAM 

Reliability,  Availability,  and  Maintainability 

RAS 

Requirements  Allocations  Sheet 

RCS 

Radar  Cross  Section 

RDT 

Reliability  Development  Testing 

RDT&E 

Research,  Development,  Test  and  Evaluation 

RFP 

Request  for  Proposal 

RGT 

Reliability  Growth  Test 

RM 

Resource  Manager 

RQT 

Reliability  Qualification  Test 

SAR 

Selected  Acquisition  Report 

SDD 

Software  Design  Document 

SDI 

Strategic  Defense  Initiative 

SDIO 

Strategic  Defense  Initiative  Organization 

SDP 

Software  Development  Plan 

SDR 

System  Design  Paper 

SECARMY 

Secretary  of  the  Army 

SECDEF 

Secretary  of  the  Defense 

SECNAV 

Secretary  of  the  Navy 

SEF 

Stability  Enhancement  Function 

SEMP 

System  Egineering  Management  Plan 

SEMS 

System  Engineering  Management  Schedule 

SFR 

Sytems  Functional  Review 

SIL 

Software  Integration  Laboratory 

SIS 

Stall  Inhibit  System 

SON 

Statement  of  Operational  Need 

SOW 

Statement  of  Work 

SPAWAR 

Space  and  Warfare 

SPEC 

Specification 

SPO 

System  Program  Office 

SRR 

Systems  Requirements  Review 

SRS 

Software  Requirement  Specification 

SSD 

Segment  Design  Document 

SSR 

Software  Specification  Review 

STA 

System  Threat  Assessment 

STAR 

System  Threat  Assessment  Report 

STEP 

Simulation,  Test  &  Evaluation  Process 

STP 

Software  Test  Plan 

SQA 

Software  Quality  Assurance 

SW 

Software 
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T&E 

Test  and  Evaluation 

TAAF 

Test,  Analyze  and  Fix 

TADS 

Theater  Air  Defense  System 

TAFT 

Test,  Analyze,  Fix  and  Test 

TEAM 

Test,  Evaluation,  Analysis,  and  Modeling 

TEC 

Test  and  Evaluation  Committee 

TECG 

T&E  Coordinating  Group 

TECHEVAL 

Technical  Evaluation  (Navy  Term) 

TECOM 

Test  and  Evaluation  Command 

TEMA 

Test  and  Evaluation  Agency 

TEMP 

Test  and  Evaluation  Master  Plan 

TEP 

Test  and  Evaluation  Plan 

TERC 

Test  and  Evaluation  Resources  Committee 

TERIB 

Test  and  Evaluation  Reliance  and  Investment  Board 

TEXCOM 

Test  and  Experimentation  Command 

TIRIC 

Training  Instrumentation  Resource  Investment  Committee 

TIWG 

Test  Integrated  Working  Group 

TLS 

Time  Line  Sheet 

TM 

Technical  Manual 

TMC 

Test  Management  Council 

TPO 

Test  Program  Outline 

TPM 

Technical  Performance  Measurement 

TPWG 

Test  Planning  Working  Group 

TR 

Test  Report 

TRADOC 

Training  and  Doctrine  Command 

TRMS 

TRADOC  Resource  Management  System 

TRP 

Test  Resources  Plan 

TRR 

Test  Readiness  Review 

TRS 

Test  Requirements  Sheet 

TSARC 

Test  Schedule  and  Review  Committee 

USAFE/DOQ 

U.S.  Air  Force-Europe/Directorate  of  Operations-Operations 

USD(A&T) 

Undersecretary  of  Defense  (Acquisition  &  Technology) 

WBS 

Work  Breakdown  Structure 

WSMR 

White  Sands  Missile  Range 
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APPENDIX  B 
DOD  GLOSSARY  OF 
TEST  TERMINOLOGY 


ACCEPTANCE  TRIALS  —  Trials  and  material  inspection  conducted  underway  by  the 
trail  board  for  ships  constructed  in  a  private  shipyard  to  determine  suitability  for  accep¬ 
tance  of  a  ship. 

ACQUISITION  —  The  conceptualization,  initiation,  design,  development,  test,  contract¬ 
ing,  production,  deployment,  logistic  support  (LS),  modification,  and  disposal  of  weapons 
and  other  systems,  supplies,  or  services  (including  construction)  to  satisfy  DoD  needs, 
intended  for  use  in  or  in  support  of  military  missions. 

ACQUISITION  CATEGORY  (ACAT)  —  ACAT I  programs  are  Major  Defense  Acquisi¬ 
tion  Programs  (MDAPs).  An  MDAP  is  defined  as  a  program  estimated  by  the  Under 
Secretary  of  Defense  (Acquisition  and  Technology)  (USD(A&T))  to  require  eventual 
expenditure  for  research,  development,  test,  and  evaluation  of  more  than  $355  million 
(fiscal  year  (FY)96  constant  dollars)  or  procurement  of  more  than  $2,135  billion  (FY96 
constant  dollars),  or  those  designated  by  the  USD(A&T)  to  be  ACAT  I.  ACAT  I  programs 
have  two  sub-categories: 

*  1.  ACAT  ID  for  which  the  Milestone  Decision  Authority  (MDA)  is  USD(A&T).  The 
"D"  refers  to  the  Defense  Acquisition  Board  (DAB),  which  advises  the  USD(A&T)  at 
major  decision  points. 

2.  ACAT  1C  for  which  the  MDA  is  the  DoD  Component  Head  or,  if  delegated,  the  DoD 
Component  Acquisition  Executive  (CAE).  The  "C"  refers  to  Component. 

The  USD(A&T)  designates  programs  as  ACAT  ID  or  ACAT  IC. 

ACAT  lA  programs  are  Major  Automated  Information  Systems  (MAISs).  A  MAIS  is 
estimated  by  the  Assistant  Secretary  of  Defense  for  Command,  Control,  Communications, 
and  Intelligence  (ASDfC^D)  to  require  program  costs  for  any  single  year  in  excess  of  $30 
million  (FY96  constant  dollars),  total  program  in  excess  of  $120  million  (FY96  constant 
dollars),  or  total  life  cycle  costs  in  excess  of  $360  million  (FY96  constant  dollars),  or  those 
designated  by  the  ASD(C^l)  to  be  ACAT  lA.  ACAT  lA  programs  have  two  sub-categories: 

1.  ACAT  I  AM  for  which  the  MDA  is  the  Office  of  the  Secretary  of  Defense  (OSD)  Chief 
Information  Officer  (CIO)  (formerly  the  Senior  IM  Official,  the  ASD(C®I)).  The  "M" 
refers  to  Major  Automated  Informatio  Systems  Review  Council  (MAISRC). 

2.  ACAT  lAC  for  which  the  MDA  is  the  DoD  Component  Chief  Information  Officer 
(CIO)  (formerly  the  Senior  IM  Official).  The  "C"  refers  to  Component.  The  ASD(C^I) 
designates  programs  as  ACAT  lAM  or  ACAT  lAC.  ACAT  II  programs  are  defined  as 
those  acquisition  programs  that  do  not  meet  the  criteria  for  an  ACAT  I  program,  but 
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do  meet  the  criteria  for  a  major  system.  A  major  system  is  defined  as  a  program 
estimated  by  the  DoD  Component  Head  to  require  eventual  expenditure  for  research, 
development,  test,  and  evaluation  of  more  than  $75M  in  1^80  constant  dollars 
(approximately  $140M  in  FY96  constant  dollars),  or  for  procurement  of  more  than 
$300M  in  FY80  constant  dollars  (approximately  $645M  in  FY96  constant  dollars),  or 
those  designated  by  the  DoD  Component  Head  to  be  ACAT II  .The  MDA  is  the  DoD 
CAE. 

ACAT  III  programs  are  defined  as  those  acquisition  programs  that  do  not  meet  the  criteria 
for  an  ACAT  I,  an  ACAT  lA,  or  an  ACAT  II.  The  MDA  is  designated  by  the  CAE  and  shall 
be  at  the  lowest  appropriate  level.  This  category  includes  less-than-major  AISs. 

ACQUISITION  DECISION  MEMORANDUM  (ADM)  —A  memorandum  signed  by  the 
milestone  decision  authority  (MDA)  that  documents  decisions  made  as  the  result  of  a 
milestone  decision  review  or  in-process  review. 

ACQUISITION  LIFE  CYCLE — The  life  of  an  acquisition  program  consists  of  phases,  each 
preceded  by  a  milestone  or  other  decision  point,  during  which  a  system  goes  through 
research,  development,  test  and  evaluation,  and  production.  Currently,  the  four  phases  are: 
(1)  Concept  Exploration  (CE)  (Phase  0);  Program  Definition  and  Risk  Reduction  (PDRR) 
(Phase  I);  (3)  Engineering  and  Manufacturing  Development  (EMD)  (Phase  II);  and  (4) 
Production,  Fielding/Deployment,  and  Operational  Support  (PF/DOS)  (Phase  III). 

ACQUISITION  PHASE  —  All  the  tasks  and  activities  needed  to  bring  a  program  to  the 
next  major  milestone  occur  during  an  acquisition  phase.  Phases  provide  a  logical  means  of 
progressively  translating  broadly  stated  mission  needs  into  well-defined  system-specific 
requirements  and  ultimately  into  operationally  effective,  suitable,  and  survivable  systems. 

ACQUISITION  PROGRAM  BASELINE  (APB)  —  A  document  that  contains  the  most 
important  cost,  schedule,  and  performance  parameters  (both  objectives  and  thresholds)  for 
the  program.  It  is  approved  by  the  Milestone  Decision  Authority  (MDA),  and  signed  by  the 
program  manager  (PM)  and  his/her  direct  chain  of  supervision,  e.g.,  for  acquisition 
category  (ACAT)  ID  programs  it  is  signed  by  the  PM,  program  executive  officer  (PEO), 
component  acquisition  executive  (CAE),  and  defense  acquisition  executive  (DAE). 

ACQUISITION  STRATEGY  —  A  business  and  technical  management  approach  de¬ 
signed  to  achieve  program  objectives  within  the  resource  constraints  imposed.  It  is  the 
framework  for  planning,  directing,  contracting  for,  and  managing  a  program.  It  provides 
a  master  schedule  for  research,  development,  test,  production,  fielding,  modification, 
postproduction  management,  and  other  activities  essential  for  program  success.  Acquisi¬ 
tion  strategy  is  the  basis  for  formulating  functional  plans  and  strategies  (e.g.,  test  and 
evaluation  master  plan  (TEMP),  acquisition  plan  (AP),  competition,  prototyping,  etc.) 

ACQUISITION  RISK  —  See  Risk. 

ADVANCED  CONCEPT  TECHNOLOGY  DEMONSTRATION  (ACTD)  —  A  means  of 
demonstrating  mature  technology  to  address  critical  military  needs.  ACTD's  themselves 
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are  not  acquisition  programs,  although  they  are  designed  to  provide  a  residual,  usable 
capability  upon  completion.  Funding  is  programmed  to  support  2  years  in  the  field. 
ACID's  are  funded  with  6.3a  (Advanced  Technology  Development  (AID))  funds. 


ADVANCED  TECHNOLOGY  DEVELOPMENT  (Budget  Category  6.3) — Projects  within 
the  6.3a  (advanced  technology  development)  program  which  are  intended  to  demonstrate 
technical  feasibility  and  maturity,  and  reduce  technical  risks  and  uncertainties  at  the 
relatively  low  costs  of  informal  processes. 


AGENCY  COMPONENT  —  A  major  organizational  subdivision  of  an  agency.  For 
example:  the  Army,  Navy,  Air  Force  and  Defense  Supply  Agency  are  agency  components 
of  the  Department  of  Defense.  The  Federal  Aviation,  Urban  Mass  Transportation  and  the 
Federal  Highway  Administrations  are  agency  components  of  the  Department  of  Transpor¬ 


tation. 


ANALYSIS  OF  ALTERNATIVES  (AOA)  —  An  analysis  of  the  estimated  costs  and 
operational  effectiveness  of  alternative  materiel  systems  to  meet  a  mission  need  and  the 
associated  program  for  acquiring  each  alternative.  Formerly  known  as  Cost  and  Opera¬ 
tional  Effectiveness  Analysis  (COEA). 

AUTOMATED  INFORMATION  SYSTEM  (AIS)  —  A  combination  of  computer  hard¬ 
ware  and  software,  data,  or  telecommunications,  that  performs  functions  such  as  collect¬ 
ing,  processing,  transmitting,  and  displaying  information.  Excluded  are  computer  re¬ 
sources,  both  hardware  and  software,  that  are  physically  part  of,  dedicated  to,  or  essential 
in  real  time  to  the  mission  performance  of  weapon  systems. 

AUTOMATIC  TEST  EQUIPMENT  (ATE)  —  An  equipment  that  is  designed  to  automati¬ 
cally  conduct  analysis  of  functional  or  static  parameters  and  to  evaluate  the  degree  of 
performance  degradation  and  perform  fault  isolation  of  unit  malfunctions. 

BASELINE — Defined  quantity  or  quality  used  as  starting  point  for  subsequent  efforts  and 
progress  measurement  that  can  be  a  technical  cost  or  schedule  baseline. 

BRASSBOARD  CONFIGURATION  —  An  experimental  device  (or  group  of  devices) 
used  to  determine  feasibility  and  to  develop  technical  and  operational  data.  It  will  normally 
be  a  model  sufficiently  hardened  for  use  outside  of  laboratory  environments  to  demon¬ 
strate  the  technical  and  operational  principles  of  immediate  interest.  It  may  resemble  the 
end  item  but  is  not  intended  for  use  as  the  end  item. 

BREADBOARD  CONFIGURATION  —  An  experimental  device  (or  group  of  devices) 
used  to  determine  feasibility  and  to  develop  technical  data.  It  will  normally  be 
configured  only  for  laboratory  use  to  demonstrate  the  technical  principles  of  immedi¬ 
ate  interest.  It  may  not  resemble  the  end  item  and  is  not  intended  for  use  as  the  projected 
end  item. 

CAPSTONE  TEST  AND  EVALUATION  MASTER  PLAN  (TEMP)  —  A  TEMP  which 
addresses  the  testing  and  evaluation  of  a  program  consisting  of  a  collection  of  individual 


B-3 


systems  which  function  collectively.  Individual  system-unique  content  requirements  are 
addressed  in  an  annex  to  the  basic  Capstone  TEMP. 


CERTIFICATION  FOR  INITIAL  OPERATIONAL  TEST  AND  EVALUATION  (lOT&E) 
— A  service  process  undertaken  in  the  engineering  and  management  development  (EMD) 
resulting  in  the  announcement  of  a  system's  readiness  to  undergo  lOT&E.  The  process 
varies  with  each  Service. 

COMPETITIVE  PROTOTYPING  STRATEGY  (CPS)  —  Prototype  competition  between 
two  or  more  contractors  in  a  comparative  side-by-side  test. 

CONCEPT  EVALUATION  PROGRAM  (CEP)  —  A  specifically  funded  Army  innovative 
testing  program.  The  CEPs  provide  commanders  and  combat  developers  a  quick  reaction 
and  simplified  process  to  resolve  combat  development,  doctrinal  and  training  issues.  In 
addition,  CEPs  solidify  combat  development  requirements  and  support  early  milestone 
decisions.  Also,  the  CEP  is  used  to  provide  an  experimental  database  for  requirements 
documents  and  to  expedite  the  materiel  acquisition  process;  however,  CEPs  are  not  to  be 
used  as  the  primary  tests  to  support  decision  review  production  decisions.  The  CEP  may 
be  conducted  at  any  time  to  support  the  concept  evaluation  (CE)  process.  Issues  satisfied 
during  the  conduct  of  a  CEP  need  not  be  examined  during  formal  operational  test  (OT)  to 
minimize  testing.  Data  from  CEPs  may  be  used  as  another  source  for  preparation  of  the 
independent  evaluation  report  (lER). 

CONCURRENCY  —  Part  of  an  acquisition  strategy  which  would  combine  or  overlap  life 
cycle  phases  (such  as  engineering  and  manufacturing  development  (EMD),  and  produc¬ 
tion),  or  activities  (such  as  development  and  operational  testing). 

CONTINGENCY  TESTING  —  Additional  testing  required  to  support  a  decision  to 
commit  added  resources  to  a  program,  when  significant  test  objectives  have  not  been  met 
during  planned  tests. 

CONTINUOUS  EVALUATION  (CE)  —  A  continuous  process,  extending  from  concept 
definition  through  deployment,  that  evaluates  the  operational  effectiveness  and  suitability 
of  a  system  by  analysis  of  all  available  data. 

COMBAT  SYSTEM  —  The  equipment,  computer  programs,  people  and  documentation 
organic  to  the  accomplishment  of  the  mission  of  an  aircraft,  surface  ship  or  submarine;  it 
excludes  the  structure,  material,  propulsion,  power  and  auxiliary  equipment,  transmis¬ 
sions  and  propulsion,  fuels  and  control  systems,  and  silencing  inherent  in  the  construction 
and  operation  of  aircraft,  surface  ships  and  submarines. 

CONFIGURATION  MANAGEMENT — The  technical  and  administrative  direction  and 
surveillance  actions  taken  to  identify  and  document  the  functional  and  physical  character¬ 
istics  of  a  configuration  item  (Cl),  to  control  changes  to  a  Cl  and  its  characteristics,  and  to 
record  and  report  change  processing  and  implementation  status.  It  provides  a  complete 
audit  trail  of  decisions  and  design  modifications. 
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CONTRACT  —  An  agreement  between  two  or  more  legally  competent  parties,  in  the 
proper  form,  on  a  legal  subject  matter  or  purpose  and  for  legal  consideration, 

CONTRACTOR  LOGISTICS  SUPPORT  —  The  performance  of  maintenance  and/or 
material  management  functions  for  a  DoD  system  by  a  commercial  activity.  Historically 
done  on  an  interim  basis  until  systems  support  could  be  transitioned  to  a  DoD  organic 
capability.  Current  policy  now  allows  for  the  provision  of  system  support  by  contractors 
on  a  long-term  basis.  Also  called  Long-Term  Contractor  Logistics  Support. 

COOPERATIVE  PROGRAMS  —  Programs  that  comprise  one  or  more  specific  coopera¬ 
tive  projects  whose  arrangements  are  defined  in  a  written  agreement  between  the  parties 
and  which  are  conducted  in  the  following  general  areas: 

1.  Research,  development,  testing,  and  evaluation  (RDT&E)  of  defense  articles 
(including  cooperative  upgrade  or  other  modification  of  a  U.S.-developed  system), 
joint  production  (including  follow-on  support)  of  a  defense  article  that  was 
developed  by  one  or  more  of  the  participants,  and  procurement  by  the  United 
States  of  a  foreign  defense  article  (including  software),  technology  (including 
manufacturing  rights),  or  service  (including  logistics  support)  that  are  imple¬ 
mented  under  Title  22  U.S.C.  §2767,  Reference  (c),  to  promote  the  rationalization, 
standardization,  and  interoperability  (RSI)  of  NATO  armed  forces  or  to  enhance 
the  ongoing  efforts  of  non-NATO  countries  to  improve  their  conventional  defense 
capabilities. 

2.  Cooperative  research  and  development  program  (R&D)  with  NATO  and  major  non- 
NATO  allies  implemented  under  Title  10  U.S.C.  §2350a,  to  improve  the  conventional 
defense  capabilities  of  NATO  and  enhance  rationalization,  standardization,  and 
interoperability  (RSI). 

3.  Data,  information,  and  personnel  exchange  activities  conducted  under  approved 
DoD  programs. 

4.  Testing  and  evaluation  (T&E)  of  conventional  defense  equipment,  munitions,  and 
technologies  developed  by  allied  and  friendly  nations  to  meet  valid  existing  U.S. 
military  requirements. 

COST  AS  AN  INDEPENDENT  VARIABLE  (CAIV)  —  Methodologies  used  to  acquire 
and  operate  affordable  DoD  systems  by  setting  aggressive,  achievable  life  cycle  cost 
objectives,  and  managing  achievement  of  these  objectives  by  trading  off  performance  and 
schedule,  as  necessary.  Cost  objectives  balance  mission  needs  with  projected  out-year 
resources,  taking  into  account  anticipated  process  improvements  in  both  DoD  and  indus¬ 
try.  CAIV  has  brought  attention  to  the  government's  responsibilities  for  setting/adjusting 
life-cycle  cost  objectives  and  for  evaluating  requirements  in  terms  of  overall  cost  conse¬ 
quences. 

CRITICAL  ISSUES  —  Those  aspects  of  a  system's  capability,  either  operational,  technical, 
or  other,  that  must  be  questioned  before  a  system's  overall  suitability  can  be  known.  Critical 
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issues  are  of  primary  importance  to  the  decision  authority  in  reaching  a  decision  to  allow 
the  system  to  advance  into  the  next  phase  of  development. 


CRITICAL  OPERATIONAL  ISSUE  (COI)  —  Operational  effectiveness  and  operational 
suitability  issues  (not  parameters,  objectives,  or  thresholds)  that  must  be  examined  in 
operational  test  and  evaluation  (OT&E)  to  determine  the  system's  capability  to  perform  its 
mission.  A  COI  is  normally  phrased  as  a  question  that  must  be  answered  in  order  to 
properly  evaluate  operational  effectiveness  (e.g.,  "Will  the  system  detect  the  threat  in  a 
combat  environment  at  adequate  range  to  allow  successful  engagement?")  or  operational 
suitability  (e.g.,  "Will  the  system  be  safe  to  operate  in  a  combat  environment?"). 

DATA  SYSTEM  —  Combinations  of  personnel  efforts,  forms,  formats,  instructions, 
procedures,  data  elements  and  related  data  codes,  communicatioirs  facilities  and  automatic 
data  processing  equipment  that  provide  an  organized  and  interconnected  means,  either 
automated,  manual  or  a  mixture  of  these  for  recording,  collecting,  processing  and  commu¬ 
nicating  data. 

DEFENSE  ACQUISITION  EXECUTIVE  (DAE)  —  The  individual  responsible  for  all 
acquisition  matters  within  the  DoD.  (See  DoDD  5000.1.) 

DEPARTMENT  OF  DEFENSE  ACQUISITION  SYSTEM  —  A  single  uniform  system 
whereby  all  equipment,  facilities,  and  services  are  planned,  designed,  developed,  acquired, 
maintained,  and  disposed  of  within  the  DoD.  The  system  encompasses  establishing  and 
enforcing  policies  and  practices  that  govern  acquisitions,  to  include  documenting  mission 
needs  and  establishing  performance  goals  and  baselines;  determining  and  prioritizing 
resource  requirements  for  acquisition  programs;  planning  and  executing  acquisition 
programs;  directing  and  controlling  the  acquisition  review  process;  developing  and 
assessing  logistics  implications;  contracting;  monitoring  the  execution  status  of  approved 
programs;  and  reporting  to  the  Congress. 

DESIGNATED  ACQUISITION  PROGRAM  —  Program  designated  by  the  Director, 
Operational  Test  and  Evaluation  or  the  Director,  Test,  Systems  Engineering,  and  Evalua¬ 
tion  for  OSD  oversight  of  test  and  evaluation. 

DEVELOPING  AGENCY  (DA)  —  The  Systems  Command  or  Chief  of  Naval  Operations 
(CNO)-designated  project  manager  assigned  responsibility  for  the  development,  test  and 
evaluation  of  a  weapon  system,  subsystem  or  item  of  equipment. 

DEVELOPMENT  TEST  AND  EVALUATION  (DT&E)—T&E  conducted  throughoutthe 
life  cycle  to  identify  potential  operational  and  technological  capabilities  and  limitations  of 
the  alternative  concepts  and  design  options  being  pursued;  support  the  identification  of 
cost-performance  tradeoffs  by  providing  analyses  of  the  capabilities  and  limitations  of 
alternatives;  support  the  identification  and  description  of  design  technical  risks;  assess 
progress  toward  meeting  critical  operational  issues  (CIOs),  mitigation  of  acquisition 
technical  risk,  achievement  of  manufacturing  process  requirements  and  system  maturity; 
assess  validity  of  assumptions  and  conclusions  from  the  analysis  of  alternatives  (AOA); 
provide  data  and  analysis  in  support  of  the  decision  to  certify  the  system  ready  for 
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operational  test  and  evaluation  (OT&E);  and  in  the  case  of  automated  information  systems, 
support  an  information  systems  security  certification  prior  to  processing  classified  or 
sensitive  data  and  ensure  a  standards  conformance  certification. 

EARLY  OPERATIONAL  ASSESSMENT  (EOA) — An  operational  assessment  conducted 
prior  to  or  in  support  of  Milestone  II. 

EFFECTIVENESS — The  extent  to  which  the  goals  of  the  system  are  attained,  or  the  degree 
to  which  a  system  can  be  elected  to  achieve  a  set  of  specific  mission  requirements. 

ENGINEERING  CHANGE — An  alteration  in  the  physical  or  functional  characteristics  of 
a  system  or  item  delivered,  to  be  delivered  or  under  development,  after  establishment  of 
such  characteristics. 

ENGINEERING  CHANGE  PROPOSAL  (ECP) —A  proposal  to  the  responsible  authority 
recommending  that  a  change  to  an  original  item  of  equipment  be  considered,  and  the 
design  or  engineering  change  be  incorporated  into  the  article  to  modify,  add  to,  delete,  or 
supersede  original  parts. 

ENGINEERING  DEVELOPMENT— The  RDTE  funding  category  that  includes  develop¬ 
ment  programs  being  engineered  for  Service  use  but  not  yet  approved  for  procurement  or 
operation.  Budget  Category  6.4  includes  those  projects  in  full-scale  development  of  Service 
use;  but  they  have  not  yet  received  approval  for  production  or  had  production  funds 
included  in  the  DoD  budget  submission  for  the  budget  or  subsequent  fiscal  year. 

EVALUATION  CRITERIA  —  Standards  by  which  achievement  of  required  technical 
and  operational  effectiveness /suitability  characteristics  or  resolution  of  technical  or 
operational  issues  may  be  evaluated.  Evaluation  criteria  should  include  quantitative 
thresholds  for  the  initial  operating  capability  (IOC)  system.  If  parameter  maturity 
grows  beyond  IOC,  intermediate  evaluation  criteria,  appropriately  time-lined,  must 
also  be  provided. 

FIRST  ARTICLE — First  article  includes  preproduction  models,  initial  production  samples, 
test  samples,  first  lots,  pilot  models,  and  pilot  lots;  and  approval  involves  testing  and 
evaluating  the  first  article  for  conformance  with  specified  contract  requirements  before  or 
in  the  initial  stage  of  production  under  a  contract. 

FIRST  ARTICLE  TESTING  (FAT)  —  Production  testing  that  is  planned,  conducted,  and 
monitored  by  the  materiel  developer.  FAT  includes  preproduction  andinitial  production 
testing  conducted  to  ensure  that  the  contractor  can  furnish  a  product  that  meets  the 
established  technical  criteria. 

FOLLOW-ON  OPERATIONAL  TEST  AND  EVALUATION  (FOT&E)  —  That  test  and 
evaluation  that  may  be  necessary  after  Milestone  III  to  refine  the  estimates  made  during 
operational  test  and  evaluation,  to  evaluate  changes  and  to  re-evaluate  the  system  to  ensure 
that  it  continues  to  meet  operational  needs  and  retains  its  effectiveness  in  a  new  environ¬ 
ment  or  against  a  new  threat. 
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FOLLOW-ON  PRODUCTION  TEST  —  A  technical  test  conducted  subsequent  to  a  full 
production  decision  on  initial  production  and  mass  production  models  to  determine 
production  conformance  for  quality  assurance  purposes.  Program  funding  category  — 
Procurement. 

FOREIGN  COMPARATIVE  TESTING  (FCT)  —  A  DoD  test  and  evaluation  program 
that  is  prescribed  in  Title  10  U.S.C.  §2350a(g),  and  is  centrally  managed  by  the  Director, 
Test,  Systems  Engineering  and  Evaluation  (DTSEE).  It  provides  funding  for  U.S.  T&E 
of  selected  equipment  items  and  technologies  developed  by  allied  countries  when  such 
items  and  technologies  are  identified  as  having  good  potential  to  satisfy  valid  DoD 
requirements. 

FUTURE-YEAR  DEFENSE  PROGRAM  (FYDP)  — (Formerly  the  Five  Year  Defense 
Program).  The  official  DoD  document  which  summarizes  forces  and  resources  associated 
with  programs  approved  by  the  Secretary  of  Defense  (SECDEF).  Its  three  parts  are  the 
organizations  affected,  appropriations  accounts  (research,  development,  test,  and  evalua¬ 
tion  (RDT&E),  operations  and  maintenance  (O&M),  etc.),  and  the  11  major  force  programs 
(strategic  forces,  airlift/sealift,  R  &  D,  etc.).  R&D  is  Program  06.  Under  the  current  planning, 
programming,  and  budgeting  system  (PPBS)  cycle,  the  FYDP  is  updated  when  the  services 
submit  their  program  objective  memorandum's  (POM's)  to  the  Office  of  the  Secretary  of 
Defense  (OSD)  (May /June),  when  the  services  submit  their  budgets  to  OSD  (Sept),  and 
when  the  President  submits  the  national  budget  to  the  Congress  (Feb).  The  primary  data 
element  in  the  FYDP  is  the  Program  Element  (PE). 

HARMONIZATION  —  Refers  to  the  process,  or  results,  of  adjusting  differences  or 
inconsistencies  in  the  qualitative  basic  military  requirements  of  the  United  States,  its  allies, 
and  other  friendly  countries.  It  implies  that  significant  features  will  be  brought  into  line  so 
as  to  make  possible  substantial  gains  in  terms  of  the  overall  objectives  of  cooperation  (e.g., 
enhanced  utilization  of  resources,  standardization,  and  compatibility  of  equipment).  It 
implies  especially  that  comparatively  minor  differences  in  "requirements"  should  not  be 
permitted  to  serve  as  a  basis  for  the  support  of  slightly  different  duplicative  programs  and 
projects. 

HUMAN  SYSTEMS  INTEGRATION — A  disciplined,  unified,  and  interactive  approach 
to  integrate  human  considerations  into  system  design  to  improve  total  system  performance 
and  reduce  costs  of  ownership.  The  major  categories  of  human  considerations  are  man¬ 
power,  personnel,  training,  human  factors  engineering,  safety,  and  health. 

INDEPENDENT  EVALUATION  REPORT  —  A  report  that  provides  an  assessment  of 
item  or  system  operational  effectiveness  and  operational  suitability  versus  critical  issues  as 
well  as  the  adequacy  of  testing  to  that  point  in  the  development  of  item  or  system. 

INDEPENDENT  OPERATIONAL  TEST  AGENCY  —  The  Army  Operational  Test  and 
Evaluation  Agency,  the  Navy  Operational  Test  and  Evaluation  Force,  the  Air  Force 
Operational  Test  and  Evaluation  Center  and  the  Marine  Corps  Operational  Test  and 
Evaluation  Agency. 
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INDEPENDENT  VERIFICATION  AND  VALIDATION  (IV&V)  —  An  independent 
review  of  the  software  product  for  functional  effectiveness  and  technical  sufficiency. 

INITIAL  OPERATIONAL  CAPABILITY  (IOC) — The  first  attainment  of  the  capability  to 
employ  effectively  a  weapon,  item  of  equipment,  or  system  of  approved  specific  character¬ 
istics  with  the  appropriate  number,  type,  and  mix  of  trained  and  equipped  personnel 
necessary  to  operate,  maintain,  and  support  the  system.  It  is  normally  defined  in  the 
operational  requirements  document  (ORD). 

INITIAL  OPERATIONAL  TEST  AND  EVALUATION  (lOT&E)  —  Operational  test 
and  evaluation  conducted  on  production,  or  production  representative  articles,  to 
determine  whether  systems  are  operationally  effective  and  suitable  for  intended  use  by 
representative  users  to  support  the  decision  to  proceed  beyond  low  rate  initial  produc¬ 
tion  (LRIP). 

IN-PROCESS  REVIEW  —  Review  of  a  project  or  program  at  critical  points  to  evaluate 
status  and  make  recommendations  to  the  decision  authority. 

INSPECTION — Visual  examination  of  the  item  (hardware  and  software)  and  associated 
descriptive  documentation  which  compares  appropriate  characteristics  with  predeter¬ 
mined  standards  to  determine  conformance  to  requirements  without  the  use  of  special 
laboratory  equipment  or  procedures. 

INTEGRATED  PRODUCT  AND  PROCESS  DEVELOPMENT  (IPPD) —A  management 
technique  that  simultaneously  integrates  all  essential  acquisition  activities  through  the  use 
of  multidisciplinary  teams  to  optimize  the  design,  manufacturing,  and  supportability 
processes.  IPPD  facilitates  meeting  cost  and  performance  objectives  from  product  concept 
through  production,  including  field  support.  One  of  the  key  IPPD  tenets  is  multidisciplinary 
teamwork  through  Integrated  Product  Teams  (IPTs). 

INTEGRATED  PRODUCT  TEAM  (IPT)  —  Team  composed  of  representatives  from  all 
appropriate  functional  disciplines  working  together  to  build  successful  programs,  identify 
and  resolve  issues,  and  make  sound  and  timely  recommendations  to  facilitate  decision 
making.  There  are  three  types  of  IPTs:  overarching  IPTs  (OIPTs)  focus  on  strategic 
guidance,  program  assessment,  and  issue  resolution;  working  IPTs  (WIPTs)  identify  and 
resolve  program  issues,  determine  program  status,  and  seek  opportunities  for  acquisition 
reform;  and  program  level  IPTs  focus  on  program  execution  and  may  include  representa¬ 
tives  from  both  government  and  after  contract  award  industry. 

INTEROPERABILITY  —  The  ability  of  systems,  units,  or  forces  to  provide  services  to  or 
accept  services  from  other  systems,  units,  or  forces  and  to  use  the  services  so  exchanged  to 
operate  effectively  together.  The  conditions  achieved  among  communications-electronics 
systems  or  items  of  communications-electronics  equipment  when  information  or  services 
can  be  exchanged  directly  and  satisfactorily  between  them  and/ or  their  users. 

ISSUES — Any  aspect  of  the  system's  capability,  either  operational,  technical  or  other,  that 
must  be  questioned  before  the  system's  overall  military  utility  can  be  known.  Operational 
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issues  are  issues  that  must  be  evaluated  considering  the  soldier  and  the  machine  as  an  entity 
to  estimate  the  operational  effectiveness  and  operational  suitability  of  the  system  in  its 
complete  user  environment. 

JOINT  DEVELOPMENT  TESTS  (JDTs) — The  JDTs  provide  information  on  intra-Service 
systems  or  equipment  requirements,  performance  or  interoperability;  on  technical  con¬ 
cepts,  requirements  or  improvements;  and  on  the  improvement  or  development  of  testing 
methodologies  or  resources. 

JOINT  OPERATIONAL  TESTS  (JOTs)  —  The  JOTs  use  actual  fielded  equipment, 
simulators  or  surrogate  equipment  in  an  exercise  or  operational  environment  to  obtain  data 
pertinent  to  inter-Service  operational  doctrine,  tactics  and  procedures. 

KEY  PERFORMANCE  PARAMETERS  —  Those  capabilities  or  characteristics  so  signifi¬ 
cant  that  failure  to  meet  the  threshold  can  be  cause  for  the  concept  or  system  selected  to  be 
reevaluated  or  the  program  to  be  reassessed  or  terminated. 

LIVE  FIRE  TEST  AND  EVALUATION  (LFT&E)  —  A  test  process  that  is  defined  in  Title 
10  U.S.C.  §2366,  that  must  be  conducted  on  a  covered  system,  major  munition  program, 
missile  program,  or  product  improvement  to  a  covered  system,  major  munition  program, 
or  missile  program  before  it  can  proceed  beyond  low  rate  initial  production  (LRIP).  A 
covered  system  is  any  vehicle,  weapon  platform,  or  conventional  weapon  system  that 
includes  features  designed  to  provide  some  degree  of  protection  to  the  user  in  combat  and 
that  is  an  acquisition  category  (ACAT)  I  or  ACAT II  program. 

LIVE  FIRE  TEST  AND  EVALUATION  REPORT  —  Report  prepared  by  the  Director, 
Operational  Test  and  Evaluation  (DOT&E)  on  survivability  and  lethality  testing.  Submit¬ 
ted  to  the  Congress  for  covered  systems  prior  to  the  decision  to  proceed  beyond  low  rate 
initial  production  (LRIP). 

LETHALITY  —  The  probability  that  weapon  effects  will  destroy  the  target  or  render  it 
neutral. 

LIFE-CYCLE  COST  —  The  total  cost  to  the  government  for  the  development,  acquisidon, 
operation  and  logistic  support  of  a  system  or  set  of  forces  over  a  defined  life  span. 

LOGISTICS  SUPPORTABILITY — The  degree  of  ease  to  which  system  design  character¬ 
istics  and  planned  logistics  resources  (including  the  logistics  support  (LS)  elements)  allow 
for  the  meeting  of  system  availability  and  wartime  usage  requirements. 

LONG  LEAD  ITEMS — Those  components  of  a  system  or  piece  of  equipment  that  take  the 
longest  time  to  procure  and,  therefore,  may  require  an  early  commitment  of  funds  in  order 
to  meet  acquisition  program  schedules. 

LOT  ACCEPTANCE  —  This  test  is  based  on  a  sampling  procedure  to  ensure  that  the 
product  retains  its  quality.  No  acceptance  or  installation  should  be  permitted  until  this  test 
for  the  lot  has  been  successfully  completed. 
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LOW  RATE  INITIAL  PRODUCTION  (LRIP) — The  minimum  number  of  systems  (other 
than  ships  and  satellites)  to  provide  production  representative  articles  for  operational  test 
and  evaluation  (OT&E),  to  establish  an  initial  production  base,  and  to  permit  an  orderly 
increase  in  the  production  rate  sufficient  to  lead  to  full-rate  production  upon  successful 
completion  of  operational  testing.  For  major  defense  acquisition  programs  (MDAPs),  LRIP 
quantities  inexcess  of  10  percent  of  the  acquisition  objective  must  be  reported  in  the  selected 
acquisition  report  (SAR).  For  ships  and  satellites  LRIP  is  the  minimum  quantity  and  rate 
that  preserves  mobilization. 

MAINTAINABILITY — The  ability  of  an  item  to  be  retained  in,  or  restored  to,  a  specified 
condition  when  maintenance  is  performed  by  personnel  having  specified  skill  levels,  using 
prescribed  procedures  and  resources,  at  each  prescribed  level  of  maintenance  and  repair. 
(See  Mean  Time  To  Repair  (MTTR).) 

MAJOR  DEFENSE  ACQUISITION  PROGRAM  —  An  acquisition  program  that  is  not  a 
highly  sensitive  classified  program  (as  determined  by  the  Secretary  of  Defense)  and  that  is 
designated  by  the  Under  Secretary  of  Defense  (Acquisition  and  Technology)  (USD(A&T)) 
as  an  MDAP,  or  estimated  by  the  USD(A&T)  to  require  an  eventual  total  expenditure  for 
research,  development,  test  and  evaluation  (RDT&E)  of  more  than 355  million  in  fiscal  year 
(FY)96  constant  dollars  or,  for  procurement,  of  more  than  2.135  billion  in  FY96  constant 
dollars. 

MAJOR  SYSTEM  (DoD)  —  A  combination  of  elements  that  shall  function  together  to 
produce  the  capabilities  required  to  fulfill  a  mission  need,  including  hardware,  equipment, 
software,  or  any  combination  thereof,  but  excluding  construction  or  other  improvements 
to  real  property.  A  system  shall  be  considered  a  major  system  if  it  is  estimated  by  the  Under 
Secretary  of  Defense  (Acquisition  and  Technology  (USD(A&T))  to  require  an  eventual  total 
expenditure  for  research,  development,  test,  and  evaluation  (RDT&E)  of  more  than  75 
million  in  FY80  constant  dollars  (approximately  140  million  in  FY96  constant  dollars),  or  for 
procurement  of  more  than  300  million  in  FY80  constant  dollars  (approximately  645  million 
in  FY96  constant  dollars). 

MAJOR  RANGE  AND  TEST  FACILITY  BASE  (MRTFB)  —  The  complex  of  major  DOD 
ranges  and  test  facilities  managed  according  to  DoD  3200.11  by  the  Director,  Test,  Systems 
Engineerting,  and  Evaluation. 

MEAN  TIME  BETWEEN  FAILURES  (MTBF)  —  For  a  particular  interval,  the  total 
functional  life  of  a  population  of  an  item  divided  by  the  total  number  of  failures  within  the 
population.  The  definition  holds  for  time,  rounds,  miles,  events,  or  other  measures  of  life 
unit.  A  basic  technical  measure  of  reliability. 

MEAN  TIME  TO  REPAIR  (MTTR)  —  The  total  elapsed  time  (clock  hours)  for  corrective 
maintenance  divided  by  the  total  number  of  corrective  maintenance  actions  during  a  given 
period  of  time.  A  basic  technical  measure  of  maintainability. 

MEASURES  OF  EFFECTIVENESS  (MOE) — A  measure  of  operational  success  that  must 
be  closely  related  to  the  objective  of  the  mission  or  operation  being  evaluated.  For  example. 
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kills  per  shot,  probability  of  kill,  effective  range,  etc.  Linkage  shall  exist  among  the  various 
MOEs  used  in  the  analysis  of  alternatives,  operations  requirements  document  (ORD)  and 
test  and  evaluation  (T&E);  in  particular,  the  MOEs,  measures  of  performance  (MOPs), 
criteria  in  the  ORD,  the  analysis  of  alternatives  ( AOA),  the  test  and  evaluation  master  plan 
(TEMP)  and  the  acquisition  program  baseline  (APB)  shall  be  consistent.  A  meaningful 
MOE  must  be  quantifiable  and  a  measure  to  what  degree  the  real  objective  is  achieved. 

MEASURES  OF  PERFORMANCE  (MOP)  —  Measures  of  lowest  level  of  performance 
representing  subsets  of  measure  of  effectiveness  (MOEs).  Examples  are  speed,  payload, 
range,  time  on  station,  frequency,  or  other  distinctly  quantifiable  performance  features. 

MILESTONE  —  The  point  when  a  recommendation  is  made  and  approval  sought 
regarding  starting  or  continuing  (proceeding  to  next  phase)  an  acquisition  program. 
Milestones  are:  0  (Approval  to  Conduct  Concept  Studies),  I  (Approval  to  Begin  a  New 
Acquisition  Program),  II  (Approval  to  Enter  Engineering  &  Manufacturing  Development 
(EMD)),  and  III  (Production  or  Fielding/Development  and  Operational  Support  (PF/DOS) 
approval.) 

MILESTONE  DECISION  AUTHORITY  (MDA)  —  The  individual  designated  in  accor¬ 
dance  with  criteria  established  by  the  Under  Secretary  of  Defense  (Acquisition  and 
Technology)  (USD(A&T)),  or  by  the  Assistant  Secretary  of  Defense  (Command,  Control, 
Communications,  and  Intelligence)  (ASD(C3I))  for  automated  information  system  (AIS) 
acquisition  programs  (DoD  5000.2-R  (Reference  C)),  to  approve  entry  of  an  acquisition 
program  into  the  next  phase. 

MILITARY  OPERATIONAL  REQUIREMENT  —  The  formal  expression  of  a  military 
need,  responses  to  which  results  in  development  or  acquisition  of  item,  equipment's,  or 
systems.  (See  Operational  Requirements  Document  (ORD).) 

MISSION  AREA  ANALYSIS  (MAA)  —  The  process  by  which  warfighting  deficiencies 
are  determined,  technological  opportunities  for  increased  system  effectiveness  and  /  or  cost 
reduction  are  assessed,  and  mission  needs  identified. 

MISSION  NEED  STATEMENT  (MNS) — A  nonsystem  specific  statement  of  operational 
capability  need  prepared  in  accordance  with  the  Chairman,  Joint  Chiefs  of  Staff  Memoran¬ 
dum  of  Policy  (lAW  CJCS  MOP)  77.  Developed  by  DoD  components  and  forwarded  to  the 
operational  validation  authority  for  validation  and  approval.  Approved  MNSs  go  to  the 
milestone  decision  authority  (MDA)  for  a  determination  on  whether  or  not  to  convene  a 
Milestone  0  review. 

MISSION  RELIABILITY  —  The  probability  that  a  system  will  perform  mission  essential 
functions  for  a  given  period  of  time  under  conditions  stated  in  the  mission  profile. 

MODEL  —  A  model  is  a  representation  of  an  actual  or  conceptual  system  that  involves 
mathematics,  logical  expressions,  or  computer  simulations  that  can  be  used  to  predict  how 
the  system  might  perform  or  survive  under  various  conditions  or  in  a  range  of  hostile 
environments. 
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MULTI-SERVICE  TEST  AND  EVALUATION  —  T&E  conducted  by  two  or  more  DoD 
components  for  systems  to  be  acquired  by  more  than  one  DoD  component,  or  for  a  DoD 
component's  systems  that  have  interfaces  with  equipment  of  another  DoD  component. 
May  be  developmental  testing  or  operational  testing  (MOT&E). 

NONDEVELOPMENT  ITEM  (NDI)  —  A  nondevelopmental  item  is  any  previously 
developed  item  of  supply  used  exclusively  for  government  purposes  by  a  Federal  Agency, 
a  State  or  local  government,  or  a  foreign  government  with  which  the  United  States  has  a 
mutual  defense  cooperation  agreement;  any  item  described  above  that  requires  only  minor 
modifications  or  modifications  of  the  type  customarily  available  in  the  commercial  market¬ 
place  in  order  to  meet  the  requirements  of  the  processing  department  or  agency. 

NONMAJOR  DEFENSE  ACQUISITION  PROGRAM  —  A  program  other  than  a  major 
defense  acquisition  program  (MDAP)  acquisition  category  (ACAT)  I  or  a  highly  sensitive 
classified  program:  i.e.,  ACAT  11  and  ACAT  111  programs. 

NUCLEAR  HARDNESS  —  A  quantitative  description  of  the  resistance  of  a  system  or 
component  to  malfunction  (temporary  and  permanent)  and/or  degraded  performance 
induced  by  a  nuclear  weapon  environment.  Measured  by  resistance  to  physical  quantities 
such  as  overpressure,  peak  velocities,  energy  absorbed,  and  electrical  stress.  Hardness  is 
achieved  through  adhering  to  appropriate  design  specifications  and  is  verified  by  one  or 
more  test  and  analysis  techniques. 

OBJECTIVE  —  The  performance  value  that  is  desired  by  the  user  and  which  the  program 
manager  (PM)  is  attempting  to  obtain.  The  objective  value  represents  an  operationally 
meaningful,  time  critical,  and  cost  effective  increment  above  the  performance  threshold  for 
each  program  parameter. 

OPEN  SYSTEMS — Acquisiton  of  Weapons  Systems  An  integrated  technical  and  business 
strategy  that  defines  key  interfaces  for  a  system  (or  a  piece  of  equipment  under  develop¬ 
ment)  in  accordance  with  those  adopted  by  formal  consensus  bodies  (recognized  industry 
standards'  bodies)  as  specifications  and  standards,  or  commonly  accepted  (de  facto) 
standards  (both  company  proprietary  and  non-proprietary)  if  they  facilitate  utilization  of 
multiple  suppliers. 

OPERATIONAL  ASSES  SMENT — An  evaluation  of  operational  effectiveness  and  opera¬ 
tional  suitability  made  by  an  independent  operational  test  activity,  with  user  support  as 
required,  on  other  than  production  systems.  The  focus  of  an  OA  is  on  significant  trends 
noted  in  development  efforts,  programmatic  voids,  areas  of  risk,  adequacy  of  require¬ 
ments,  and  the  ability  of  the  program  to  support  adequate  operational  testing  (OT).  OA 
may  be  made  at  any  time  using  technology  demonstrators,  prototypes,  mock-ups, 
engineering  development  models,  or  simulations  but  will  not  substitute  for  the  inde¬ 
pendent  operational  test  and  evaluation  (OT&E)  necessary  to  support  full  production 
decisions. 

OPERATIONAL  AVAILABILITY  (AO)  —  The  degree  (expressed  in  terms  of  1.0  or  100 
percent  as  the  highest)  to  which  one  can  expect  an  equipment  or  weapon  systems  to  work 
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properly  when  it  is  required.  The  equation  is  uptime  over  uptime  plus  downtime, 
expressed  as  Ao.  It  is  the  quantitative  link  between  readiness  objectives  and  support- 
ability. 

OPERATIONAL  EFFECTIVENESS  —  The  overall  degree  of  mission  accomplishment  of 
a  system  when  used  by  representative  personnel  in  the  environment  planned  or  expected 
(e.g.  natural,  electronic,  threat,  etc.)  for  operational  employment  of  the  system  considering 
organization,  doctrine,  tactics,  survivability,  vulnerability  and  threat  (including  counter¬ 
measures;  initial  nuclear  weapons  effects;  and  nuclear,  biological  and  chemical  contamina¬ 
tion  (NBCC)  threats). 

OPERATIONAL  EVALUATION  —  Addresses  the  effectiveness  and  suitability  of  the 
weapons,  equipment  or  munitions  for  use  in  combat  by  typical  military  users  and  the 
system  operational  issues  and  criteria;  provides  information  to  estimate  organizational 
structure,  personnel  requirements,  doctrine,  training  and  tactics;  identifies  any  operational 
deficiencies  and  the  need  for  any  modifications;  and  assesses  MANPRINT  (safety,  health 
hazards,  human  factors,  manpower  and  personnel)  aspects  of  the  system  in  a  realistic 
operational  environment. 

OPERATIONAL  REQUIREMENTS  —  User-or  user  representative-generated  validated 
needs  developed  to  address  mission  area  deficiencies,  evolving  threats,  emerging  tech¬ 
nologies  or  weapon  system  cost  improvements.  Operational  requirements  form  the  foun¬ 
dation  for  weapon  system  unique  specifications  and  contract  requirements. 

OPERATIONAL  REQUIREMENTS  DOCUMENT  (ORD) — Documents  the  users  objec¬ 
tives  and  minimum  acceptable  requirements  for  operational  performance  of  a  proposed 
concept  or  system.  Format  is  contained  in  Appendix  II,  DoD  5000.2-R. 

OPERATIONAL  SUITABILITY  —  The  degree  to  which  a  system  can  be  placed  satisfac¬ 
torily  in  field  use  with  consideration  being  given  to  availability,  compatibility,  transport¬ 
ability,  interoperability,  reliability,  wartime  usage  rates,  maintainability,  safety,  human 
factors,  manpower  supportability,  logistic  supportability,  natural  environmental  effects 
and  impacts,  documentation,  and  training  requirements. 

OPERATIONAL  TEST  AND  EVALUATION — The  field  test,  under  realistic  conditions, 
of  any  item  (or  key  component)  of  weapons,  equipment,  or  munitions  for  the  purpose  of 
determining  the  effectiveness  and  suitability  of  the  weapons,  equipment,  or  munitions  for 
use  in  combat  by  t5q?ical  military  users;  and  the  evaluation  of  the  results  of  such  tests. 

OPERATIONAL  TEST  CRITERIA — Expressions  of  the  operational  level  of  performance 
required  of  the  military  system  to  demonstrate  operational  effectiveness  for  given  func¬ 
tions  during  each  operational  test.  The  expression  consists  of  the  function  addressed,  the 
basis  for  comparison,  the  performance  required  and  the  confidence  level. 

OPERATIONAL  TEST  READINESS  REVIEW  (OTRR)  —  A  review  to  identify  problems 
that  may  impact  the  conduct  of  an  OT&E.  The  OTRRs  are  conducted  to  determine  changes 
required  in  planning,  resources  or  testing  necessary  to  proceed  with  the  OT&E.  Participants 
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include  the  operational  tester  (chair),  evaluator,  material  developer,  user  representative, 
logisticians,  HQDA  staff  elements  and  others  as  necessary. 

PARAMETER — A  determining  factor  or  characteristic.  Usually  related  to  performance  in 
developing  a  system. 

PERFORMANCE  —  Those  operational  and  support  characteristics  of  the  system  that 
allow  it  to  effectively  and  efficiently  perform  its  assigned  mission  over  time.  The  support 
characteristics  of  the  system  include  both  supportability  aspects  of  the  design  and  the 
support  elements  necessary  for  system  operation. 

PILOT  PRODUCTION  —  Production  line  normally  established  during  engineering  and 
management  development  (EMD)  to  test  new  manufacturing  methods  and  procedures. 
Normally  funded  by  research,  development,  test,  and  evaluation  (RDT&E)  until  the  line  is 
proven. 

POSTPRODUCTION  TESTING  —  Testing  conducted  to  assure  that  materiel  that  is 
reworked,  repaired,  renovated,  rebuilt  or  overhauled  after  initial  issue  and  deployment 
conforms  to  specified  quality,  reliability,  safety  and  operational  performance  standards. 
Included  in  postproduction  tests  are  surveillance  tests,  stockpile  reliability  and  recondi¬ 
tioning  tests. 

PREPLANNED  PRODUCT  IMPROVEMENT  (P’l)  —  Planned  future  evolutionary  im¬ 
provement  of  developmental  systems  for  which  design  considerations  are  effected  during 
development  to  enhance  future  application  of  projected  technology.  Includes  improve¬ 
ments  planned  for  ongoing  systems  that  go  beyond  the  current  performance  envelope  to 
achieve  a  needed  operational  capability. 

PREPRODUCTION  PROTOTYPE  —  An  article  in  final  form  employing  standard  parts, 
representative  of  articles  to  be  produced  subsequently  in  a  production  line. 

PREPRODUCTION  QUALIFICATION  TEST— The  formal  contractual  tests  that  ensure 
design  integrity  over  the  specified  operational  and  environmental  range.  These  tests 
usually  use  prototype  or  preproduction  hardware  fabricated  to  the  proposed  production 
design  specifications  and  drawings.  Such  tests  include  contractual  reliability  and  maintain¬ 
ability  (R&M)  demonstrahons  tests  required  prior  to  production  release. 

PROBABILITY  OF  KILL  (Pk)  —  The  lethality  of  a  weapon  system.  Generally  refers  to 
armaments,  (e.g.,  missiles,  ordnance,  etc.).  Usually  the  statistical  probability  that  the 
weapon  will  detonate  close  enough  to  the  target  with  enough  effectiveness  to  disable  the 
target. 

PRODUCT  IMPROVEMENT  (PI)  —  Effort  to  incorporate  a  configuration  change  involv¬ 
ing  engineering  and  testing  effort  on  end  items  and  depot  repairable  components,  or 
changes  on  other  than  developmental  items  to  increase  system  or  combat  effectiveness  or 
extend  useful  military  life.  Usually  results  from  feedback  from  the  users. 


B-15 


PRODUCTION  ACCEPTANCE  TEST  AND  EVALUATION  (PAT&E)  —  Test  and 
evaluation  of  production  items  to  demonstrate  that  items  procured  fulfill  requirements  and 
specifications  of  the  procuring  contract  or  agreements. 

PRODUCTION  ARTICLE  —  The  end  item  under  initial  or  full  rate  production. 

PRODUCTION  PROVEOUT  —  A  technical  test  conducted  prior  to  production  testing 
with  prototype  hardware  to  determine  the  most  appropriate  design  alternative.  This 
testing  may  also  provide  data  on  safety,  the  achieveability  of  critical  system  technical 
characteristics,  refinement  and  ruggedization  of  hardware  configurations,  and  determina¬ 
tion  of  technical  risks. 

PRODUCTION  QUALIFICATION  TEST  (PQT) — A  technical  test  completed  prior  to  the 
full  rate  production  decision  to  ensure  the  effectiveness  of  the  manufacturing  process, 
equipment,  and  procedures.  This  testing  also  serves  the  purpose  of  providing  data  for  the 
independent  evaluation  required  for  materiel  release  so  that  the  evaluator  can  address  the 
adequacy  of  the  materiel  with  respect  to  the  stated  requirements.  These  test  are  conducted 
on  a  number  of  samples  taken  at  random  from  the  first  production  lot,  and  are  repeated  if 
the  process  or  design  is  changed  significantly,  and  when  a  second  or  alternative  source  is 
brought  on  line. 

PROGRAM  MANAGER  (PM)  —  A  military  or  civilian  official  who  is  responsible  for 
managing,  through  integrated  product  teams  (IPTs),  an  acquisition  program. 

PROTOTYPE  —  An  original  or  model  on  which  a  later  system/item  is  formed  or  based. 
Early  prototypes  may  be  built  during  program  definition  and  risk  reduction  (PDRR)  phase 
and  tested  prior  to  milestone  (MS)  II  decision.  Selected  protot)q)ing  may  continue  in  phase 
II,  Engineering  and  Manufacturing  Development  (EMD),  as  required  to  identify  and 
resolve  specific  design  and  manufacturing  risks  early  in  the  phase  or  in  support  of 
preplanned  product  improvement  (P^I)  or  evolutionary  acquisition  (EA). 

QUALIFICATIONS  TESTING  —  Simulates  defined  operational  environmental  condi¬ 
tions  with  a  predetermined  safety  factor,  the  results  indicating  whether  a  given  design  can 
perform  its  function  within  the  simulated  operational  environment  of  a  system. 

QUALITY  ASSURANCE  —  A  planned  and  systematic  pattern  of  all  actions  necessary  to 
provide  confidence  that  adequate  technical  requirements  are  established,  that  products 
and  services  conform  to  established  technical  requirements,  and  that  satisfactory  perfor¬ 
mance  is  achieved. 

REALISTIC  TEST  ENVIRONMENT  —  The  conditions  under  which  the  system  is 
expected  to  be  operated  and  maintained,  including  the  natural  weather  and  climatic 
conditions,  terrain  effects,  battlefield  disturbances,  and  enemy  threat  conditions. 

RELIABILITY — The  ability  of  a  system  and  its  parts  to  perform  its  mission  without  failure, 
degradation,  or  demand  on  the  support  system.  (See  MeanTime  Between  Failures  (MTBF).) 
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REQUIRED  OPERATIONAL  CHARACTERISTICS  —  System  parameters  that  are  pri¬ 
mary  indicators  of  the  system's  capability  to  be  employed  to  perform  the  required  mission 
functions,  and  to  be  supported. 

REQUIRED  TECHNICAL  CHARACTERISTICS  —  System  parameters  selected  as  pri¬ 
mary  indicators  of  achievement  of  engineering  goals.  These  need  not  be  direct  measures  of, 
but  should  always  relate  to  the  system's  capability  to  perform  the  required  mission 
functions,  and  to  be  supported. 

RESEARCH  —  1.  Systematic  inquiry  into  a  subject  in  order  to  discover  or  revise  facts, 
theories,  etc.  to  investigate.  2.  Means  of  developing  new  technology  for  potential  use  in 
defense  systems. 

RISK  —  A  measure  of  the  inability  to  achieve  program  objectives  within  defined  cost  and 
schedule  constraints.  Risk  is  associated  with  all  aspects  of  the  program,  e.g.,  threat,  technology, 
design  processes.  Work  breakdown  structure  (WBS)  elements,  etc.  It  has  two  components: 

The  probability  of  failing  to  achieve  a  particular  outcome;  and 

The  consequences  of  failing  to  achieve  that  outcome. 

RISK  ASSESSMENT  —  The  process  of  identifying  program  risks  within  risk  areas  and 
critical  technical  processes,  analyzing  them  for  their  consequences  and  probabilities  of 
occurrence,  and  prioritizing  them  for  handling. 

RISK  MONITORING  —  A  process  that  systematically  tracks  and  evaluates  the  perfor¬ 
mance  of  risk  items  against  established  metrics  throughout  the  acquisition  process  and 
develops  further  risk  reduction  handling  options  as  appropriate. 

SAFETY  —  The  application  of  engineering  and  management  principles,  criteria,  and 
techniques  to  optimize  safety  within  the  constraints  of  operational  effectiveness,  time,  and 
cost  throughout  all  phases  of  the  system  life  cycle. 

SAFETY/HEALTH  VERIFICATION  —  The  development  of  data  used  to  evaluate  the 
safety  and  health  features  of  a  system  to  determine  its  acceptability.  This  is  done  primarily 
during  developmental  test  (DT)  and  user  or  operational  test  (OT)  and  evaluation  and 
supplemented  by  analysis  and  independent  evaluations. 

SAFETY  RELEASE  —  A  formal  document  issued  to  a  user  test  organization  before  any 
hands-on  use  or  maintenance  by  personnel.  The  safety  release  indicates  the  system  is  safe 
for  use  and  maintenance  by  typical  user  personnel  and  describes  the  system  safety 
analyses.  Operational  limits  and  precautions  are  included.  The  test  agency  uses  the  data  to 
integrate  safety  into  test  controls  and  procedures  and  to  determine  if  the  test  objectives  can 
be  met  within  these  limits. 

SELECTED  ACQUISITION  REPORT  —  Standard,  comprehensive,  summary  status 
reports  on  major  defense  acquisition  programs  (MDAPs)  (acquisition  category  (ACAT)  I) 
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required  for  periodic  submission  to  the  Congress.  They  include  key  cost,  schedule,  and 
technical  information. 

SIMULATION  —  A  simulation  is  a  method  for  implementing  a  model.  It  is  the  process  of 
conducting  experiments  with  a  model  for  the  purpose  of  understanding  the  behavior  of  the 
system  modeled  under  selected  conditions  or  of  evaluating  various  strategies  for  the 
operation  of  the  system  within  the  limits  imposed  by  developmental  or  operational  criteria. 
Simulation  may  include  the  use  of  analog  or  digital  devices,  laboratory  models,  or 
"testbed"  sites.  Simulations  are  usually  programmed  for  solution  on  a  computer;  however, 
in  the  broadest  sense,  military  exercises,  and  wargames  are  also  simulations. 

SIMULATOR  —  A  generic  term  used  to  describe  equipment  used  to  represent  weapon 
systems  in  development  testing,  operational  testing,  and  training,  e.g.,  a  threat  simulator 
has  one  or  more  characteristics  which,  when  detected  by  human  senses  or  man-made 
sensors,  provide  the  appearance  of  an  actual  threat  weapon  system  with  a  prescribed 
degree  of  fidelity. 

SPECIFICATION — A  document  used  in  development  and  procurement  which  describes 
the  technical  requirements  for  items,  materials,  and  services,  including  the  procedures  by 
which  it  will  be  determined  that  the  requirements  have  been  met.  Specifications  may  be 
unique  to  a  specific  program  (program-peculiar)  or  they  may  be  common  to  several 
applications  (general  in  nature). 

SUBTEST  —  An  element  of  a  test  program.  A  subset  is  a  test  conducted  for  a  specific 
purpose  (e.g.,  rain,  dust,  transportability,  missile  firing,  fording). 

SURVIVABILITY  —  Survivability  The  capability  of  a  system  and  its  crew  to  avoid  or 
withstand  a  man-made  hostile  environment  without  suffering  an  abortive  impairment  of 
its  ability  to  accomplish  its  designated  mission. 

SUSCEPTIBILITY — The  degree  to  which  a  device,  equipment,  or  weapon  system  is  open 
to  effective  attack  due  to  one  or  more  inherent  weaknesses.  Susceptibility  is  a  function  of 
operational  tactics,  countermeasures,  probability  of  enemy  fielding  a  threat,  etc.  Suscepti¬ 
bility  is  considered  a  subset  of  survivability. 

SYSTEM — 1.  The  organization  of  hardware,  software,  material,  facilities,  personnel,  data, 
and  services  needed  to  perform  a  designated  function  with  specified  results,  such  as  the 
gathering  of  specified  data,  its  processing,  and  delivery  to  users.  2.  A  combination  of  two 
or  more  interrelated  equipment's  (sets)  arranged  in  a  functional  package  to  perform  an 
operational  function  or  to  satisfy  a  requirement. 

SYSTEM  ENGINEERING,  DEFENSE  —  A  comprehensive,  iterative  technical  manage¬ 
ment  process  that  includes  translating  operational  requirements  into  configured  systems, 
integrating  the  technical  inputs  of  the  entire  design  team,  managing  interfaces,  character¬ 
izing  and  managing  technical  risk,  transitioning  technology  from  the  technology  base  into 
program  specific  efforts,  and  verifying  that  designs  meet  operational  needs.  It  is  a  life  cycle 
activity  that  demands  a  concurrent  approach  to  both  product  and  process  development. 


SYSTEM  ENGINEERING  PROCESS  —  A  logical  sequence  of  activities  and  decisions 
transforming  an  operational  need  into  a  description  of  system  performance  parameters  and 
a  preferred  system  configuration. 

SYSTEM  SPECIFICATION — States  all  necessary  functional  requirements  of  a  system  in 
terms  of  technical  performance  and  mission  requirements,  including  test  provisions  to 
assure  that  all  requirements  are  achieved.  Essential  physical  constraints  are  included. 
System  specifications  state  the  technical  and  mission  requirements  of  the  system  as  an 
entity. 

SYSTEM  THREAT  ASSESSMENT  (STA) — Describes  the  threat  to  be  countered  and  the 
projected  threat  environment.  The  threat  information  must  be  validated  by  the  Defense 
Intelligence  Agency  (DIA)  programs  reviewed  by  the  Defense  Acquisition  Board  (DAB). 

TECHNICAL  EVALUATION  —  The  study,  investigations,  or  test  and  evaluation  (T&E) 
by  a  developing  agency  to  determine  the  technical  suitability  of  materiel,  equipment,  or  a 
system,  for  use  in  the  military  services.  (See  Development  Test  and  Evaluation  (DT&E).) 

TECHNICAL  FEASIBILITY  TEST — A  technical  test  conducted  post-MS  0  and  pre-MS  I 
or  MS  I/II  (under  the  Army  Streamlined  Acquisition  Process)  to  assist  in  determining 
safety  and  establishing  system  performance  specifications  and  feasibility. 

TECHNICAL  PERFORMANCE  MEASUREMENT  (TPM)  —  Describes  all  the  activities 
undertaken  by  the  government  to  obtain  design  status  beyond  that  treating  schedule  and 
cost.  A  TPM  manager  is  defined  as  the  product  design  assessment  which  estimates,  through 
tests  the  values  of  essential  performance  parameters  of  the  current  design  of  work 
breakdown  structure  (WBS)  product  elements.  It  forecasts  the  values  to  be  achieved 
through  the  planned  technical  program  effort,  measures  differences  between  achieved 
values  and  those  allocated  to  the  product  element  by  the  system  engineering  process,  and 
determines  the  impact  of  these  differences  on  system  effectiveness. 

TECHNICAL  TESTER  —  The  command  or  agency  that  plans,  conducts  and  reports  the 
results  of  Army  technical  testing.  Associated  contractors  may  perform  development  testing 
on  behalf  of  the  command  or  agency. 

TECHNICAL  TESTS — A  generic  Army  term  for  testing  that  gathers  technical  data  during 
development  testing,  technical  feasibility  testing,  qualification  testing,  joint  development 
testing  and  contractor/foreign  testing.  Soldier  operator-maintainer  test  and  evaluation 
personnel  are  used  during  technical  testing  when  appropriate. 

TEST — Any  program  or  procedure  which  is  designed  to  obtain,  verify,  or  provide  data  for 
the  evaluation  of  research  and  development  (R&D)  (other  than  laboratory  experiments); 
progress  in  accomplishing  development  objectives;  or  performance  and  operational  capa¬ 
bility  of  systems,  subsystems,  components,  and  equipment  items. 

TEST  AND  EVALUATION  (T&E)  —  Process  by  which  a  system  or  components  provide 
information  regarding  risk  and  risk  mitigation  and  empirical  data  to  validate  models  and 
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simulations.  T&E  permit,  as  assessment  of  the  attainment  of  technical  performance, 
specifications  and  system  maturity  to  determine  whether  systems  are  operationally  effec¬ 
tive,  suitable  and  survivable  for  intended  use.  There  are  two  types  of  T&E-Developmental 
(DT&E)  and  Operational  (OT&E).  (See  Operational  Test  and  Evaluation  (OTE&E),  Initial 
Operational  Test  and  Evaluation  (lOT&E),  and  Developmental  Test  and  Evaluation 
(DT&E).) 

TEST  AND  EVALUATION  MASTER  PLAN  —  Documents  the  overall  structure  and 
objectives  of  the  test  and  evaluation  (T&E)  program.  It  provides  a  framework  within  which 
to  generate  detailed  T&E  plans  and  it  documents  schedule  and  resource  implications 
associated  with  the  T&E  program.  The  TEMP  identifies  the  necessary  developmental  test 
and  evaluation  (DT&E),  operational  test  and  evaluation  (OT&E)  and  live  fire  test  and 
evaluation  (LFT&E)  activities.  It  relates  program  schedule,  test  management  strategy  and 
structure,  and  required  resources  to:  critical  operational  issues  (COIs);  critical  technical 
parameters;  objectives  and  thresholds  documented  in  the  Operational  Requirements 
Document  (ORD);  evaluation  criteria;  and  (5)  milestone  decision  points.  For  multiservice 
or  joint  programs,  a  single  integrated  TEMP  is  required.  Component-unique  content 
requirements,  particularly  evaluation  criteria  associated  with  COIs,  can  be  addressed  in  a 
component-prepared  annex  to  the  basic  TEMP.  (See  Capstone  TEMP).) 

TEST-BEDS  —  A  system  representation  consisting  of  actual  hardware  and/or  software 
and  computer  models  or  prototype  hardware  and/or  software 

TEST  CRITERIA  —  Standards  by  which  test  results  and  outcome  are  judged. 

TEST  DESIGN  PLAN  —  A  statement  of  the  conditions  under  which  the  test  is  to  be 
conducted,  the  data  required  from  the  test  and  the  data  handling  required  to  relate  the  data 
results  to  the  test  conditions. 

TEST  INSTRUMENTATION  —  Test  instrumentation  is  scientific,  automated  data  pro¬ 
cessing  equipment  (ADPE),  or  technical  equipment  used  to  measure,  sense,  record, 
transmit,  process  or  display  data  during  tests,  evaluations  or  examination  of  materiel, 
training  concepts  or  tactical  doctrine.  Audio-visual  is  included  as  instrumentation  when 
used  to  support  Army  testing. 

TEST  RESOURCES  —  A  collective  term  that  encompasses  all  elements  necessary  to  plan, 
conduct  and  collect/analyze  data  from  a  test  event  or  program.  Elements  include  test 
funding  and  support  manpower  (including  TDY  costs),  test  assets  (or  units  under  test),  test 
asset  support  equipment,  technical  data,  simulation  models,  test-beds,  threat  simulators, 
surrogates  and  replicas,  special  instrumentation  peculiar  to  a  given  test  asset  or  test  event, 
targets,  tracking  and  data  acquisition,  instrumentation,  equipment  for  data  reduction, 
communications,  meteorology,  utilities,  photography,  calibration,  security,  recovery, 
maintenance  and  repair,  frequency  management  and  control  and  base/facility  support 
services. 

THREAT  — The  sum  of  the  potential  strengths,  capabilities,  and  strategic  objectives  of  any 
adversary  that  can  limit  or  negate  U.S.  mission  accomplishment  or  reduce  force,  system,  or 
equipment  effectiveness. 
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THRESHOLDS  —  The  minimum  acceptable  value  which,  in  the  user's  judgment,  is 
necessary  to  satisfy  the  need.  If  threshold  values  are  not  achieved,  program  performance 
is  seriously  degraded,  the  program  may  be  too  costly,  or  the  program  may  no  longer  be 
timely. 

TRANSPORTABILITY — The  capability  of  materiel  to  be  moved  by  towing,  self-propul¬ 
sion,  or  carrier  through  any  means,  such  as  railways,  highways,  waterways,  pipelines, 
oceans,  and  airways.  (Full  consideration  of  available  and  projected  transportation  assets, 
mobility  plans  and  schedules,  and  the  impact  of  system  equipment  and  support  items  on 
the  strategic  mobility  of  operating  military  forces  is  required  to  achieve  this  capability.) 

UNKNOWN-UNKNOWNS  (UNK(s))  —  Future  situation  impossible  to  plan,  predict,  or 
even  know  what  to  look  for. 

USER — An  operational  command  or  agency  that  receives  or  will  receive  benefit  from  the 
acquired  system.  Commanders-in-Chief  (CINCs)  and  the  Services  are  the  users.  There  may 
be  more  than  one  user  for  a  system.  The  Services  are  seen  as  users  for  systems  required  to 
organize,  equip,  and  train  forces  for  the  CINCs  of  the  unified  command. 

USER  FRIENDLY  —  Primarily  a  term  used  in  automated  date  processing  (ADP),  it 
connotes  a  machine  (hardware)  or  program  (software)  that  are  compatible  with  a  person's 
ability  to  operate  them  successfully  and  easily. 

USER  REPRESENTATIVES  —  A  command  or  agency  that  has  been  formally  designated 
by  proper  authority  to  represent  single  or  multiple  users  in  the  requirements  and  acquisi¬ 
tion  process.  The  Services  and  the  Service  components  of  the  Commanders-in-Chief 
(CINCs)  are  normally  the  user  representative.  There  should  be  only  one  user  representative 
for  a  system. 

VALIDATION  —  1.  The  process  by  which  the  contractor  (or  as  otherwise  directed  by  the 
DoD  component  procuring  activity)  tests  a  publication/ technical  manual  (TM)  for  techni¬ 
cal  accuracy  and  adequacy.  2.  The  procedure  of  comparing  input  and  output  against  an 
edited  file  and  evaluating  the  result  of  the  comparison  by  means  of  a  decision  table 
established  as  a  standard. 

VARIANCE  (Statistical)  —  A  measure  of  the  degree  of  spread  among  a  set  of  values;  a 
measure  of  the  tendency  of  individual  values  to  vary  from  the  mean  value.  It  is  computed 
by  subtracting  the  mean  value  from  each  value,  squaring  each  of  these  differences, 
summing  these  results,  and  dividing  this  sum  by  the  number  of  values  in  order  to  obtain 
the  arithmetic  mean  of  these  squares. 

VULNERABILITY  —  The  characteristics  of  a  system  that  cause  it  to  suffer  a  definite 
degradation  (loss  or  reduction  of  capability  to  perform  the  designated  mission)  as  a  result 
of  having  been  subjected  to  a  certain  (defined)  level  of  effects  in  an  unnatural  (man-made) 
hostile  environment.  Vulnerability  is  considered  a  subset  of  survivability. 
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WORK  BREAKDOWN  STRUCTURE  (WBS)  —  An  organized  method  to  break  down  a 
project  into  logical  subdivisions  or  subprojects  at  lower  and  lower  levels  of  details.  It  is  very 
useful  in  organizing  a  project. 

WORKING-LEVEL  INTEGRATED  PRODUCT  TEAM  (WIPT)  —  Team  of  representa¬ 
tives  from  all  appropriate  functional  disciplines  working  together  to  build  successful  and 
balanced  programs,  identify  and  resolve  issues,  and  make  sound  and  timely  decisions. 
WIPTs  may  include  members  from  both  government  and  industry,  including  program 
contractors  and  sub-contractors.  A  committee,  which  includes  non-govemment  represen¬ 
tatives,  to  provide  an  industry  view,  would  be  an  advisory  committee  covered  by  Federal 
Advisory  Committee  Act  (FACA)  and  must  follow  the  procedures  of  that  Act. 
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APPENDIX  C 

TEST  RELATED  DATA  ITEM  DESCRIPTIONS 


extracted  from  DoD  5010.12-L, 
Acquisition  Management  System  and 
Data  Requirement  Control  List  (AMSDL) 


Acceptance  Test  Plan 

Airborne  Sound  Measurements  Test  Report 
Airframe  Rigidity  Test  Report 
Ammunition  Test  Expenditure  Report 
Armor  Material  Test  Reports 
Ballistic  Acceptance  Test  Report 
C.P.  Propeller  Test  Agenda 
Coordinated  Test  Plan 
Corrosion  Testing  Reports 
Damage  Tolerance  Test  Results  Reports 
Demonstration  Test 
Plan 
Report 

Directed  Energy  Survivability  Test  Plan 
Durability  Test  Results  Report 
Electromagnetic  Compatibility  Test  Plan 
Electromagnetic  Interference  Test 
Plan 
Report 

Electrostatic  Discharge  Sensitivity  Test  Report 
Emission  Control  (EMCON)  Test  Report 
Endurance  Test  (EMCS)  Failure  Reports 
Engineer  Design  Test  Plan 
Environmental  Design  Test  Plan 
Environmental  Test  Report 
Equipment  Test  Plan  (Nonsystem) 

Factory  Test 
Plan 

EMCS  Plan 
EMCS  Procedures 
EMCS  Reports 

First  Article  Qualification  Test  Plan 
Flight  Flutter  Test  Report 
Flutter  Model  Test  Report 

Hardware  Diagnostic  Test  System  Development  Plan 
High-Impact  Shock  Test  Procedures 
Hull  Test  Results  (Boats)  Report 
Human  Engineering  Test 
Plan 
Report 


DI-QCIC-80154, 80553 

DI-HFAC-80272 

DI-T-30734 

DI-MISC-80060 

DI-MISC-80073 

DI-MISC-80246 

UDI-T-23737 

DI-MGMT-80937 

DI-MFFP-80108 

DI-T-30725 

DI-QCIC-80775 

DI-QCIC-80774 

DI-R-1786 

DI-T-30726 

DI-T-3704B 

DI-EMCS-80201 

DI-EMCS-80200 

DI-RELI-80670 

DI-R-2059 

DI-ATrS-80366 

DI-MGMT-80688 

DI-ENVR-80861 

DI-ENVR-80863 

DI-T-3709A 

DI-QCIC-80153 

DI-ATTS-80360 

DI-ATTS-80361 

DI-ATTS-80362 

DI-T-5315A 

DI-T-30733 

DI-T-30732 

DI-ATTS-80005 

DI-ENVR-80709 

UDI-T-23718 

DI-HFAC-80743 

DI-HFAC-80744 


C-1 


Inspection  and  Test  Plan 

DI-QCIC-81110 

Installation  Test 

Plan 

DI-QCIC-80155 

Procedures 

DI-QCIC-80511 

Report 

DI-QCIC-80140, 80512 

Integrated  Circuit  Test  Documentation 

DI-ATTS-80888 

Maintainability /Testability  Demonstration  Test 

Plan 

DI-MNTY-80831 

Report 

DI-MNTY-80832 

Maintenance  Training  Equipment  Test  Outlines 

DI-H-6129A 

Master  Test  Plan/Program  Test  Plan 

DI-T-30714 

NBC  Contamination  Survivability  Test  Plan 

DI-R-1779 

Nuclear  Survivability  Test 

Plan 

DI-NUOR-80928 

Report 

DI-NUOR-80929 

Packaging  Test 

Plan 

DI-PACK-80456 

Report 

DI-PACK-80457 

Part,  Component,  or  Subsystem  Test  Plan(s) 

DI-MISC-80759 

Parts  (Non-standard)  Test  Data  Report 

DI-MISC-81058 

Parts  Qualification  Test  Plan 

DI-T-5477A 

Performance  Oriented  Packaging  Test  Report 

DI-PACK-81059 

Production  Test 

Plan 

DI-MNTY-80173 

Report 

DI-NDTI-80492 

Quality  Conformance  Test  Procedures 

DI-RELI-80322 

Radar  Spectrum  Management  (RSM)  Test  Plan 

DI-MISC-81113 

Randomizer  Test  Report 

DI-NDTI-80884 

Reliability  Test 

Plan 

DI-RELI-80250 

Procedures 

DI-RELI-80251 

Reports 

DI-RELI-80252 

Research  and  Development  Test  and  Acceptance  Plan 

DI-T-30744 

Rough  Handling  Test  Report 

DI-T-5144C 

Ship  Acceptance  Test  (SAT) 

Schedule 

DI-T-23959B 

Report 

DI-T-23190A 

Shipboard  Industrial  Test  Procedures 

DI-QCIC-80206 

Shock  Test 

Extension  Request 

DI-ENVR-80706 

Report 

DI-ENVR-80708 

Software  General  Unit  Test  Plan 

DI-MCCR-80307 

Software  Test 

Description 

DI-MCCR-80015A 

Plan 

DI-MCCR-80014A 

Procedures 

DI-MCCR-80310 

Report 

DI-MCCR-80017A,  80311 
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Software  System 

Devel  Test  and  Eval  Plan 
Integration  and  Test  Plan 
Sound  Test  Failure  Notif  and  Recomm 
Special  Test  Equipment  Plan 
Spectrum  Signature  Test  Plan 
Static  Test 
Plan 
Reports 

Structureborne  Vibration  Accel  Measurement  Test 
Superimposed  Load  Test  Report 
Tempest  Test 
Request 
Plan 

Test  Change  Proposal 
Test  Elements  List 

Test  Facility  Requirements  Document  (TFRD) 

Test  Package 
Test 

Plan 

Plans/Procedures 

Procedure 

Procedures 

Test  Plan  Documentation  for  AIS 
Test  Program 

Documentation  (TPD) 

Integration  Logbook 
TPS  and  OTPS  Acceptance  Test 
Procedures  (ATPS) 

Report  (ATR) 

Test  Reports 


DI-MCCR-80309 

DI-MCCR-80308 

DI-HFAC-80271 

DI-T-30702 

DI-R-2068 

DI-T-21463A 

DI-T-21464A 

DI-HFAC-80274 

DI-T-5463A 

DI-EMCS-80218 

DI-T-1912A 

DI-T-26391B 

DI-QCIC-80204 

DI-FACR-80810 

DI-ILSS-81085 

DI-NDTI-80566 

DI-NDTI-80808 

DI-NDTI-80603 

UDI-T-23732B 

DI-IPSC-80697 

DI-ATTS-80284 

DI-ATTS-80281 

DI-ATTS-80282A 

DI-ATTS-80283A 

DI-NDTI-80809A, 

DI-MISC-80653 


Test  Requirements  Document 


DI-T-2181, 

DI-ATTS-80002, 80041 


Test  Scheduling  Report 
Testability 

Program  Plan 
Analysis  Report 

Trainer  Test  Procedures  and  Results  Report 
Vibration  and  Noise  Test  Reports 
Vibration  Testing 
Extension 
Report 

Welding  Procedure  Qualification  Test  Report 


Dl-MISC-80761 

DI-T-7198 

DI-T-7199 

DI-T-25594C 

DI-T-30735 

UDI-T-23752 

UDl-T-23762 

DI-MISC-80876 
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standardization  Areas 


Lead  Service 


ATTS  Automatic  Test  Technology  Standards  10 

CMAN  Configuration  Management  SD 

E  Engineering  and  Configuration  Documentation  * 

EMCS  Electromagnetic  Compatibility  EC 

ENVR  Environmental  Requirements  and  Related  Test  Meth  TE 

FACR  Facility  Construction  Design  Requirements  YD 

GDRQ  General  Design  Requirements  SD 

H  Human  Factors  * 

HFAC  Human  Factors  Ml 

ILSS  Integrated  Logistics  Support  Standards  WS 

IPSC  Information  Processing  Standards  for  Computers  02 

MCCR  Mission  Critical  Computer  Resources  10 

MFFP  Metal  Finishes  and  Finishing  Processes  and  Proc  MR 

MGMT  Management  VAR 

MISC  Miscellaneous  SD 

MNTY  Maintainability  17 

NDTI  Nondestructive  Testing  and  Inspection  MR 

NUOR  Nuclear  Ordnance  DS 

PACK  Packing,  Packaging,  Preservation  and  Transport  SM 

QCIC  Quality  Control/Assurance  and  Inspection  AR 

R  Related  Design  Requirements  * 

RELI  Reliability  17 


C-4 


Standardization  Areas 


Lead  Service 


S 

System/Subsystem  Analysis 

SAFT 

Safety 

10 

TMSS 

Technical  Manual  Specifications  and  Standards 

TM 

T 

Test 

>*• 

V 

Test 

* 

♦ 

UDI 

prior  to  1  Jul  1985;  being  attritted  out 

indicates  DID  unique  to  originator;  being  attritted  out 

Lead  Services 


02 

USAF 

ACS  Information  Systems 

10 

USAF 

AFMC  Command  Standardization  Office 

17 

USAF 

AFMC  Rome  Air  Development  Center 

AR 

USA 

AMC  AMCCOM  ARDEC 

DS 

DNA 

Defense  Nuclear  Agency 

EC 

USN 

SPAWAR 

MI 

USA 

AMC  MICOM 

MR 

USA 

AMC  ARL  Matl  Tech  Lab 

SD 

OSD 

Standardization  and  Data  Management 

SM 

USA 

AMC  Packing,  Storage  and  Containerization  Center 

TE 

USA 

AMCTECOM 

TM 

USA 

AMC  Matl  Readiness  Spt  Activity 

WS 

OSD 

Wpn  Spt  Improvement  and  Analysis  Office 

YD 

USN 

Nav  Fac  Engr  Cmd 
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Development  Testing 

Contractor 

Government 

Limited  Procurement  Programs 

Responsibilities 

Support  of  Technical  Reviews/Milestone  Decisions 

Test  Program  Integration 
Director,  Operational  Test  and  Evaluation 
DoD  Test  Process 
DT&E  and  the  Review  Process 
Early  Operational  Assessments 
EC/CnSR  Testing 
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5-3 

2-1, 2-2 
3-9 
3-11 
3-9 
25-1 
3-5 
3-7 
3-6 
5-5 
24-1 
9-1 
9-1 

20-2, 25-10 
23-1 
2-2 
9-3 
8-7 
3-1 

4-4, 4-6, 4-7, 4-8, 7-5 
1-3, 12-1 
4-5, 15-2 
11-4, 13-4 
4-2, 12-4, 13-7 
4-6, 7-7 
7-9, 7-10 
7-8 

2-10, 4-5, 8-1 
6-1, 9-1 
7-1 
7-5 
7-6 
7-9 

7- 5 

8- 1 
4-4, 7-6 

3-4 

2-9 

8-1 

1-5, 6-4, 11-2, 12-1, 12-3 
20-1 
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End  of  Test  Phase  Report 
Evaluation  Agencies 
Evaluation 

Criteria 

Development/Operational  Test  and  Evaluation 

Differences  between  T&E 

Issues 

Plan 

Planning 

Process 

Operational 

Technical 

Evolutionary  Acquisition 
First  Article  Test 
Follow-on  Operational  T&E 
Foreign  Comparative  Test  Program 
Formal  Reviews 

Systems  Requirements  Review 
Systems  Functional  Review 
Software  Specification  Review 
Preliminary  Design  Review 
Critical  Design  Review 
Test  Readiness  Review 
Formal  Qualification  Review 
Production  Readiness  Review 
Functional  Configuration  Audit 
Hazardous  Weapons  Testing 
Human  Factors  Evaluation 
Independent  Verification  and  Validation 
Initial  OT&E 
Integrated  Test  Plan 
Logistics  Support  Planning 
Objectives 

Logistics  Support  T&E 
Data  Collection  and  Analyses 
Limitations 
Use  of  Results 
Support  Package 
Integrated  Test  Approach 
Introduction  to  Test  &  Evaluation  Process 
Issues,  Critical 
Joint  T&E 

Laser  Weapons,  Testing 
Life  Cycle  T&E 
Life  Testing 

Live  Fire  Test  Guidelines 
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3-4, 4-6, 11-2, 11-4, 13-9 
13-1 
13-4 
13-7 
13-1 
13-4 
5-7, 13-2 
13-6 
13-1 
13-9 
13-9 

16-2, 20-6 
10-2 

1-7, 6-7, 11-3, 12-6 
3-2, 3-4, 22-1 
2-9, 2-10, 8-1 
8-4 
8-4 
8-4 
8-4 
8-6 
8-6 
8-6 
8-7 
8-6 
24-1 
11-2, 19-4 
17-8 

1-7, 2-2, 6-4, 6-7, 11-3, 12-5 
7-6, 16-8, 20-1 
19-1 
19-1 
19-2 
13-6, 19-7 
19-7 
19-7 

19- 7 

20- 1 
2-1 

11-4, 13-4 
6-8 
24-3 

2-1, 2-4, 7-1, 12-1 
7-7 

6-8, 6-9, 18-1, 18-3 
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Logistics  T&E 

19-1 

Planning 

19-2 

Beyond  Low  Rate  Initial  Production  Report 

3-2,3-5,5-12,10-3 

Low  Rate  Initial  Production 

10-3 

Major  Range  and  Test  Facility  Base 

3-4, 15-2 

Marine  Corps  Acquisition  Executive 

3-11 

Marine  Corps  DT&E  Organization 

3-11 

Marine  Corps  Operational  Test  &  Evaluation  Agency 

3-11 

Missile  System  T&E 

25-5 

Major  Milestone  Reviews 

1-3, 5-3 

Matrix  Analysis  Techniques 

13-7 

NATO  Comparative  Test  Program 

22-2 

Navy  DT&E  Organizations 

3-7 

Navy  Operational  Test  &  Evaluation  Force 

3-9 

NDl  Testing 

23-1 

Nondevelopment  Items  Definition 

23-1 

Nuclear  Hardness  T&E 

6-10 

Nuclear,  Biological  &  Chemical  Weapons  T&E 

6-9, 24-1 

Objectives  and  Thresholds 

2-7, 5-3, 13-5 

Operational  T&E 

6-3, 7-3, 11-1 

OPEVAL 

11-3 

OT&E  to  Support  Milestone  Decisions 

12-1 

OT&E  Contractor  Responsibilities 

4-7, 4-8 

OSD  Oversight 

3-2 

Physical  Configuration  Audit 

8-6 

Policy,  The  Congress 

3-1 

OSD 

3-2 

Army 

3-5 

Navy 

3-7 

Air  Force 

3-9 

Marine  Corps 

3-11 

Planning  Guidelines  for  Logistic  T&E 

19-1 

Post  Production  T&E 

1-7, 7-4, 10-3, 12-6 

Production  Qualification  Tests 

6-1, 10-2 

Product  Baseline  and  T&E 

2-9, 5-5, 8-5 

Production  Acceptance  Test  &  Evaluation 

7-4, 10-3 

Production  Readiness  Reviews 

10-2, 10-5 

Program  Definition  and  Risk  Reduction  Phase 

1-5, 2-2, 7-1, 12-3 

Program  Office  Responsibility  for  T&E 

4-1 

Qualification  Operational  T&E 

11-3 

Qualification  Testing 

10-2 

Quick-Look  Report 

5-9, 11-6 

Realism  during  Testing 

6-4, 1-5 

Reliability,  Availability  &  Maintainability  Assessments 

7-8, 19-5 

Reliability  Development  Testing 

7-8, 19-6 

Reports 

3-1, 5-9, 5-11, 11-6, 13-9, 18-6 

E-3 


Requirements  Documentation 

5-1 

Resources 

4-11, 15-1 

Risk  Management 

1-1, 1-8, 2-7 

Schedules  Formulation 

4-3, 16-4 

Security  and  T&E 

24-5 

Service  Resource 

15-1 

Planning 

15-4 

Service  Test  Facilities 

6-1, 15-4 

Army 

15-4 

Navy 

15-5 

Air  Force 

15-7 

Ship  System  T&E 

25-13 

Simulation,  Test,  and  Evaluation  Process  (STEP) 

6-1, 14-7 

Software  Development  Process 

17-4 

Software  Life  Cycle  T&E 

17-4 

Space  Systems  Testing 

7-9, 24-3 

Specifications 

4-6, 8-4 

Surface  Vehicle  T&E 

25-16 

System  Life  Cycle 

2-1 

Systems  Engineering  Process 

2-6 

Technical  Management  Planning 

2-7, 5-1 

Technical  Reviews  and  Audits 

4-5, 8-1 

Test  Concepts,  lOT&E 

3-5, 11-5 

Test  and  Evaluation  Master  Plan 

3-4, 5-7, 15-1, 16-1 

Test  Design  Plan 

5-7 

Test  Funding 

3-5, 4-5, 15-8 

Technical  Performance  Measurement 

2-7 

Test  Plan 

5-9 

Test  Program  Integration 

7-6, 16-8, 20-1 

Test  Reports 

3-1, 5-9, 11-6, 13-9, 18-6 

Testing  with  Limitations 

24-1 

Thresholds 

2-7, 5-3, 13-5 

Transition  to  Production 

10-3 

Transition  Planning 

10-3 

Testing  during  Transition 

10-3 

Test  Resource  Planning 

15-1 

Testing  for  Nuclear  Hardness  and  Survivability 

6-10, 18-6 

TEMP  Requirements 

16-1 

Upgrades,  Enhancements,  Modification  T&E 

1-7, 2-4, 4-10, 7-4, 12-6 

Validity  of  Modeling  and  Simulation 

14-4 

Warranties,  Impact  on  T&E 

7-9 
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APPENDIX  F 
POINTS  OF  CONTACT 

SERVICE  TEST  AND  EVALUATION  COURSES 

ARMY 

Commander  US  Army  Test  and  Evaluation  Command 
ATTN:  AMSTE-TE 

Aberdeen  Proving  Ground,  MD  21005-5055 

Commander  US  Army  Logistics  Management  Group 
ATTN:  AMXMC-ACM-MA 
Fort  Lee,  VA  23801-6048 

Commander  US  Army  Operational  Test  &  Evaluation  Command 
ATTN:  CSTE-ZA 
45-1  Ford  Avenue 
Alexandria,  VA  22302 

NAVY 

Commander  Space  &  Naval  Warfare  Systems  Command 
ATTN:  Management  Operations 
National  Center,  Building  1 
Washington,  DC  203361 

Commander  Operational  Test  &  Evaluation  Force 
ATTN:  Dest02B 
Norfolk,  VA  23511-6388 


AIR  FORCE 


Commander  Air  Force  Operational  Test  &  Evaluation  Center 
ATTN:  Asst.  Director,  Training 
Building  20130 

Kirtland  Air  Force  Base,  New  Mexico  87117-7001 

Commander  Air  Force  Institute  of  Technology 
ATTN:  Student  Operations 
Wright-Patterson  Air  Force  Base,  OH  45433 


OSD 


Defense  Test  &  Evaluation  Professional  Institute 
Naval  Air  Warfare  Center,  Weapons  Division 
Code02PI 

Point  Mugu,  CA  933042 
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